The stereo 2015 / flow 2015 / scene flow 2015 benchmark consists of 200 training scenes and 200 test scenes (4 color images per scene, saved in loss less png format). Compared to the stereo 2012 and flow 2012 benchmarks, it comprises dynamic scenes for which the ground truth has been established in a semi-automatic process. Our evaluation server computes the percentage of bad pixels averaged over all ground truth pixels of all 200 test images. For this benchmark, we consider a pixel to be correctly estimated if the disparity or flow end-point error is <3px or <5% (for scene flow this criterion needs to be fulfilled for both disparity maps and the flow map). We require that all methods use the same parameter set for all test pairs. Our development kit provides details about the data format as well as MATLAB / C++ utility functions for reading and writing disparity maps and flow fields. More details can be found in Object Scene Flow for Autonomous Vehicles (CVPR 2015).
Our evaluation table ranks all methods according to the number of erroneous pixels. All methods providing less than 100 % density have been interpolated using simple background interpolation as explained in the corresponding header file in the development kit. Legend:
D1: Percentage of stereo disparity outliers in first frame
D2: Percentage of stereo disparity outliers in second frame
Fl: Percentage of optical flow outliers
SF: Percentage of scene flow outliers (=outliers in either D0, D1 or Fl)
bg: Percentage of outliers averaged only over background regions
fg: Percentage of outliers averaged only over foreground regions
all: Percentage of outliers averaged over all ground truth pixels
Note: On 13.03.2017 we have fixed several small errors in the flow (noc+occ) ground truth of the dynamic foreground objects and manually verified all images for correctness by warping them according to the ground truth. As a consequence, all error numbers have decreased slightly. Please download the devkit and the annotations with the improved ground truth for the training set again if you have downloaded the files prior to 13.03.2017 and consider reporting these new number in all future publications. The last leaderboards before these corrections can be found here (optical flow 2015) and here (scene flow 2015). The leaderboards for the KITTI 2015 stereo benchmarks did not change.
Additional information used by the methods
Stereo: Method uses left and right (stereo) images
Multiview: Method uses more than 2 temporally adjacent images
Motion stereo: Method uses epipolar geometry for computing optical flow
Additional training data: Use of additional data sources for training (see details)
HCI/Bosch Robust Vision Challenge: Optical flow and stereo vision challenge on high resolution imagery recorded at a high frame rate under diverse weather conditions (e.g., sunny, cloudy, rainy). The Robert Bosch AG provides a prize for the best performing method.
Image Sequence Analysis Test Site (EISATS): Synthetic image sequences with ground truth information provided by UoA and Daimler AG. Some of the images come with 3D range sensor information.
Middlebury Stereo Evaluation: The classic stereo evaluation benchmark, featuring four test images in version 2 of the benchmark, with very accurate ground truth from a structured light system. 38 image pairs are provided in total.
Daimler Stereo Dataset: Stereo bad weather highway scenes with partial ground truth for freespace
Make3D Range Image Data: Images with small-resolution ground truth used to learn and evaluate depth from single monocular images.
Middlebury Optical Flow Evaluation: The classic optical flow evaluation benchmark, featuring eight test images, with very accurate ground truth from a shape from UV light pattern system. 24 image pairs are provided in total.
Citation
When using this dataset in your research, we will be happy if you cite us:
@ARTICLE{Menze2018JPRS,
author = {Moritz Menze and Christian Heipke and Andreas Geiger},
title = {Object Scene Flow}, journal = {ISPRS Journal of Photogrammetry and Remote Sensing (JPRS)},
year = {2018}
}
@INPROCEEDINGS{Menze2015ISA,
author = {Moritz Menze and Christian Heipke and Andreas Geiger},
title = {Joint 3D Estimation of Vehicles and Scene Flow}, booktitle = {ISPRS Workshop on Image Sequence Analysis (ISA)},
year = {2015}
}
Stereo: Method uses left and right (stereo) images
Multiview: Method uses more than 2 temporally adjacent images
Motion stereo: Method uses epipolar geometry for computing optical flow
Additional training data: Use of additional data sources for training (see details)
