DiVa-360: The Dynamic Visual Dataset for Immersive Neural Fields

Paper

Code

Dataset

Abstract

Advances in neural fields are enablling high-fidelity capture of shape and appearance of dynamic 3D scenes. However, this capbabilities lag behind those offered by conventional representations such as 2D videos because of algorithmic challenges and the lack of large-scale multi-view real-world datasets. We address the dataset limitations with DiVa-360, a real-world 360° dynamic visual dataset that contains synchronized high-resolution and long-duration multi-view video sequences of table-scale scenes captured using a customized low-cost system with 53 cameras. It contains 21 object-centric sequences categorized by different motion types, 25 intricate hand-object interaction sequences, and 8 long-duration sequences for a total of 17.4M frames. In addition, we provide foreground-background segmentation masks, synchronized audio, and text descriptions. We benchmark the state-of-the-art dynamic neural field methods on DiVa-360 and provide insights about existing methods and future challenges on long-duration neural field capture.

Downloading Data

Setup

We store our dataset on Globus, so to download the data to your local machine, we use Globus Command Line Interface (Globus-CLI) and Globus Connect Personal (GCP). You will first need to install GCP. Follow these intructions depending on your machine: Mac, Windows, Linux.

Note: When you are installing GCP, you will have to name your collection/endpoint. You are free to name it however you choose, but we suggest naming it “<name> personal machine”.

Next, you will need to install Globus-CLI and login. Run the following commands:

$ pip install globus-cli
$ globus login

This will take you to a login page. You can either log in through a listed institution, through any Google account, or through an ORCID iD. After logging in, you will see a terms of service page. To continue, click “Allow”. To download the data, you will be copying the data from the DiVa360 endpoint to the endpoint you just created on your local machine when you installed GCP. First, setup the DiVa360 endpoint:

$ diva360_ep=8ac249c5-8d25-4faa-9247-745d0213c615

Next, setup your personal endpoint for your local machine:

$ globus endpoint local-id
$ personal_ep=<output of the above command>

Important Note: To transfer to your GCP endpoint, the GCP software must be running and connected for the transfer to complete. However, you can close your terminal after the transfer has started.

Download

Downloading raw data (note that this is a total of 1.4 TB of data):

# Download all raw data
$ globus transfer $diva360_ep:/raw_data/ $personal_ep:<path to destination> --recursive
$ globus transfer $diva360_ep:/raw_data_long/ $personal_ep:<path to destination> --recursive
# Downloading a single sequence
$ globus transfer $diva360_ep:/raw_data/synced/2023-05-02_session_<sequence>_synced.tar.gz $personal_ep:<path to destination>/2023-05-02_session_<sequence>_synced.tar.gz
$ globus transfer $diva360_ep:/raw_data_long/synced/2023-10-21_session_<sequence>_synced.tar.gz $personal_ep:<path to destination>/2023-10-21_session_<sequence>_synced.tar.gz

Downloading processed data (note that this is a total of 1.8 TB of data):

# Download all processed data
$ globus transfer $diva360_ep:/processed_data/ $personal_ep:<path to destination> --recursive
$ globus transfer $diva360_ep:/processed_data_long/ $personal_ep:<path to destination> --recursive
# Downloading a single sequence
$ globus transfer $diva360_ep:/processed_data/<sequence>/ $personal_ep:<path to destination> --recursive
$ globus transfer $diva360_ep:/processed_data_long/<sequence>/ $personal_ep:<path to destination> --recursive

Downloading trained models (note that this is a total of 6 TB of data):

$ globus transfer $diva360_ep:/model_data/ $personal_ep:<path to destination> --recursive
$ globus transfer $diva360_ep:/model_data_long/ $personal_ep:<path to destination> --recursive
$ globus transfer $diva360_ep:/model_data_exp/ $personal_ep:<path to destination> --recursive

Downloading rendered videos (note that this is a total of 63.1 GB of data):

$ globus transfer $diva360_ep:/all_videos/ $personal_ep:<path to destination> --recursive

You can also transfer multiple files or folders at once using the batch transfer feature. Here is an example:

$ globus transfer --batch batch_transfer.txt $diva360_ep $personal_ep

Assuming you have the following batch_transfer.txt file:

# to copy a file
<path to file> <path to destination>/<file name>
# to copy a folder
<path to folder> <path to destination>/<folder name> --recursive

To check the status of your transfer, use the following command:

globus task show <task ID>

Sequences

Dynamic Data Examples

Note: These videos have sound

Dynamic Baseline Examples

Object Centric Sequence

Note: This video has sound

Hand-Object Interaction Sequence

Note: This video has sound

Long Duration Sequence

Note: This video has sound

Citations

@inproceedings{diva360,
    title={DiVa-360: The Dynamic Visual Dataset for Immersive Neural Fields},
    author={Cheng-You Lu and Peisen Zhou and Angela Xing and Chandradeep Pokhariya and Arnab Dey and Ishaan N Shah and Rugved Mavidipalli and Dylan Hu and Andrew Comport and Kefan Chen and Srinath Sridhar},
    booktitle = {Conference on Computer Vision and Pattern Recognition 2024},
    year={2024}
}

Acknowledgements

This work was supported by NSF grants CAREER #2143576 and CNS #2038897, ONR grant N00014-22-1-259, ONR DURIP grant N00014-23-1-2804, a gift from Meta Reality Labs, an AWS Cloud Credits award, and NSF CloudBank. Arnab Dey was supported by H2020 COFUND program BoostUrCareer under Marie SklodowskaCurie grant agreement #847581. We thank George Konidaris, Stefanie Tellex, and Rohith Agaram.

Contact

Cheng-You Lu (cheng-you.lu@student.uts.edu.au)

Peisen Zhou (peisen_zhou@alumni.brown.edu)

Angela Xing (angela_xing@brown.edu)