StreetAware Dataset

This README describes a high-resolution audio, video, and LiDAR dataset of three urban intersections in Brooklyn, New York, totaling approximately 8 unique hours. The data is collected with custom Reconfigurable Environmental Intelligence Platform (REIP) sensors that are designed with the ability to accurately synchronize multiple video and audio streams.

Code repository: https://github.com/reip-project/street-aware

Description and methods

We collected this data in August 2022. For each recording session, sensors where placed with focus on individual crosswalks of the intersection (with two sensors on each side of the road).

We collected data at the following intersections:

  1. Chase : 40.693541, -73.984277 120 Myrtle Ave, Brooklyn, NY 11201.
  2. Park: 40.697455, -73.980519 50 Navy St, Brooklyn, NY 10016.
  3. Dumbo: 40.702321, -73.993141 46 Old Fulton St, Brooklyn, NY 11201.

The REIP sensors were placed at positions indicated by encircled numbers in corresponding <location>_map.pdf files. Colors of the circles denote different recording sessions.

Folder structure:

Sub-folders and files repeat for the sensor and LiDAR data, for each intersection and session, as shown in chase_3 below.

├── chase_1                             *intersection and session number*
├── chase_2
├── chase_3                               
│   ├── sensor_1.zip                    *sensor number*
│   │   ├── array.json                  *audio metadata*
│   │   ├── array.wav                   *processed audio*
│   │   ├── left_half.mp4               *left camera video at half resolution*
│   │   ├── left.json                   *left camera metadata*
│   │   ├── left.mp4                    *left camera video at full resolution*
│   │   ├── left_people.json            *pose and face detections with bounding boxes and scores for left camera*
│   │   ├── left_quarter.mp4            *left camera video at quarter resolution*
│   │   ├── right_half.mp4              *right camera video at half resolution*
│   │   ├── right.json                  *right camera metadata*
│   │   ├── right.mp4                   *right camera video at full resolution*
│   │   ├── right_people.json           *pose and face detections with bounding boxes and scores for right camera*
│   │   └── right_quarter.mp4           *right camera video at quarter resolution*
│   ├── sensor_2.zip
│   ├── sensor_3.zip
│   ├── sensor_4.zip
│   ├── mosaic.mp4                      *synchronized mosaic video*
│   ├── stereo.zip                      *stereo sound from sensors in sync with mosaic.mp4*
│   │   ├── stereo_1.wav                *sensor_1 audio (channels 0 and 3)*
│   │   ├── street_2.wav
│   │   ├── street_3.wav
│   │   └── street_4.wav
│   └── lidar.zip                       *raw lidar data*
│       ├── 0.npy                       *first full revolution LiDAR scan (at 20 Hz)*
│       ├── 0.json                      *metadata for the first full revolution scan*
│       ...
├── chase_map.pdf                       *overhead map of intersection with sensor positions*
├── dumbo_1
├── dumbo_2
├── dumbo_3
├── dumbo_4
├── dumbo_map.pdf
├── park_1
├── park_2
├── park_3
├── park_4
├── park_map.pdf
├── reip_sensor.jpg                      *image of the REIP sensor*
└── README.md                            *this file*

Attention! Known issues/limitations:

  1. Lidar data is not synchronized with audio and video data;
  2. Park_4 recording session has lots of mising data because of copying error;
  3. Dumbo_4 session experienced lots of lost frames because of sensor overheating.

Preferred citation

Piadyk, Y., Rulff, J., Brewer, E., Hosseini, M., Ozbay, K., Sankar, M., & Chakradharand, S. (2023). StreetAware: A High-Resolution Synchronized Multimodal Urban Scene Dataset. New York University. https://doi.org/10.58153/q1byv-qc065