Overview

Our dataset is organized using a hierarchical structure, illustrated in the diagram below. The names in square brackets are abbreviations used in codes and file names.

Data structure

Split

A split defines a subset of data for training, validation, or testing. In the testing split, only images are publicly accessible, while annotations are kept for online evaluation.

View

A view is a specific direction in which a camera points. We provide 6 camera views: Front, Left/Right 45°, Left/Right 90°, and one Front (Left) view. Please note that the Front and Front (Left) views can be paired for stereo vision. Further, we also provide one LiDAR view named Center view.

Data group

In each of the six views, we have different data groups. The data group is the smallest unit for download. Each data group contains one specific type of data or annotations:

• Pixel-level data group. The data groups for images and pixel-level labels (i.e., semantic masks, depth maps, and optical flow) are zip/tar files that pack the data of this type in driving sequences.
• Object-level data group (marked by light green in the above diagram). The data groups for object labels (i.e., 2D object labels, 3D object labels, and instance labels) are provided in Scalabel formatted json files. Please find more information below.

Sequence

A sequence is a 50s-long driving record of consecutive frames. The sequence ID is a unique 8-digit hexadecimal number. Each sequence is a folder of frames for a pixel-level data group, where the folder is named by its sequence ID.

Frame

A frame contains the data captured at a specific timestamp in sequences. It is the minimal granularity in our dataset. Files are named with the pattern of <frame_number>_<data_group>_<view>.<ext>. Here, <data_group> and <view> are abbreviations for the corresponding data group and view; and <ext> is the file extension. The file extension and format for each pixel-level data group can be found in the following table.

Data conversion

By using the following commands, you can convert the downloaded data into other formats. Please refer to SHIFT DevKit for more information.

Sensors

Camera intrinsics

All RGB cameras have the same intrinsic parameters. Our RGB cameras have resolution of 1280 * 800 pixel and field of view (FoV) of 90°. Their intrinsic parameters are (center_x, center_y) = (640, 400), (focal_x, focal_y) = (640, 640), skew = 0.

The baseline of the stereo pair, i.e., horizontal gap between the Front and the Front Left cameras, is 0.2m.

Poses of views to the ego vehicle

The relative poses of the camera views with respect to the ego vehicle are defined in sensors.yaml, where location denotes the offset on x, y and z axes from the ego vehicle’s origin (in meters) and rotation denotes the angle of rotation with Euler angles (in degrees). Please refer to the figure below.

Annotations

Scalabel format

We provide labels for 2D/3D object detection, 2D/3D multiple object tracking (MOT), and instance segmentation using the Scalabel format. Below is the format description.

Dataset:
- frames[ ]:                           // storing all frames from every sequences
    - frame0001: Frame
    - frame0002: Frame
    ...
- config:
    - image_size:                       // all images have the same size
        - width: 1280
        - height: 800
    - categories: ["car", "truck", ...] // define the categories of objects

Frame:
- name: string                          // e.g., "abcd-1234/00000001_img_center.png"
- videoName: string                     // e.g., "abcd-1234", unique across whole dataset
- attributes                            // for discrete domain shifts        
    - timeofday_coarse: 
        {"daytime" | "dawn/dusk" | "night"} 
    - weather_coarse: 
        {"clear" | "cloudy" | "overcast" | "rainy" | "foggy }
    - timeofday_fine: 
        {"noon" | "daytime" | "morning/afternoon" | "dawn/dusk" | "sunrise/sunset" | 
        "night" | "dark night"} 
    - weather_fine:
        {"clear" | "slight cloudy" | "partial cloudy" | "overcast" | "small rain" | 
         "mid rain" | "heavy rain" | "small fog" | "heavy fog" } 
    - town:                             // name of the town
        {"01" | "02" | "03" | "04" | "05" | "06" | "07" | "10HD"}                              
    - sun_altitude_angle                // [-5, 90] degrees
    - cloudiness                        // [0, 100] percentage
    - precipitation                     // [0, 100] percentage
    - precipitation_deposits            // [0, 100] percentage
    - wind_intensity                    // [0, 100] percentage
    - sun_azimuth_angle                 // [0, 360] degrees
    - fog_density                       // [0, 100] percentage
    - fog_distance                      // meters
    - wetness                           // [0, 100] percentage
    - fog_falloff                       // meters
- intrinsics                            // intrinsic parameters (only for cameras)
    - focal: [x, y]                     // in pixel
    - center: [x, y]                    // in pixel
- extrinsics:                           // extrinsics parameters
    - location: [x, y, z]               // in meter
    - rotation: [rot_x, rot_y, rot_z]   // in radius (XYZ Euler angle)
- timestamp: int                        // time in this video, ms
- frameIndex: int                       // frame index in this video
- size:
    - width: 1280
    - height: 800
- labels [ ]:
    - id: string                         // for tracking, unique in current sequence
    - index: int
    - category: string                   // classification
    - attributes
        - ego: bool                      // if the 3D bounding box is the ego vehicle (only for LiDAR)
    - box2d:                             // 2D bounding box (in pixel)
        - x1: float
        - y1: float
        - x2: float
        - y2: float
    - box3d:                              // 3D bounding box (in ego coordinate system)  
        - orientation: [rot_x, rot_y, rot_z]    // Euler angle, in radius
        - location: [x, y, z]                   // in meter
        - dimension: [height, width, length]    // in meter
    - rle:                                // instance mask in RLE format, for instance segmentation
        - counts: [int]
        - size: (height, width)          

Sequence information

The sequence information files summarize the environment attributes of each sequence with tabular (.csv) format. Below are column definitions.

Segmentation labels

We provide semantic segmentation labels following Carla’s semantic labels. Below is a table of the 23 classes in the ground truth and their correspondence to Cityscapes that can be used in semantic segmentation experiments. We highly recommend users follow Cityscapes’s evaluation protocol, where certain classes must be ignored (defined in the last column of the table).

LiDAR point clouds

LiDAR point clouds are stored in binary (.ply) format, where each point has 4 properties (x, y, z, intensity) in metric space. A Gaussian-based noise model was applied to the pointclouds along the direction of rays used in Carla.

We would like to bring to users attention that the provided pointclouds are NOT sensitive to domain shift (e.g., rain, fog, …), as the CARLA LiDAR simulation is not affected by different environmental conditions. As such, the pointclouds should not be used for domain adaptation studies of LiDAR-specific behaviors. Some reasonable applications can be 3D detection/tracking under the same domain, or usage as ground truth.

Coordinate systems and extrinsics

We follow KITTI’s convention for image- and LiDAR-based perception tasks. To see how to use extrinsics, please refer to the script sensor_pose.py.

Visualization

Below is a visualization of point clouds’ tracking using Scalabel web tools.

How to use SHIFT?

Our dataset aims to foster research in several under-explored fields for autonomous driving that are crucial to safety. We enumerate some of the possible use cases of our dataset:

• Robustness and generality: Investigating how a perception systems’ performance degrades at increasing levels of domain shift.
• Uncertainty estimation and calibration: Assessing and developing uncertainty estimation methods that work under realistic domain shifts.
• Multi-task perception system: Studying the combination of tasks and developing multi-task models to effectively counteract domain shift.
• Continual learning: Investigating how to utilize domain shifts incrementally, e.g., continual domain adaptation and curriculum learning.
• Test-time learning: Developing and evaluating learning algorithms for continuously changing environments, e.g., test-time adaptation.