Scene Flow Evaluation 2015


The stereo 2015 / flow 2015 / scene flow 2015 benchmark consists of 200 training scenes and 200 test scenes (4 color images per scene, saved in loss less png format). Compared to the stereo 2012 and flow 2012 benchmarks, it comprises dynamic scenes for which the ground truth has been established in a semi-automatic process. Our evaluation server computes the percentage of bad pixels averaged over all ground truth pixels of all 200 test images. For this benchmark, we consider a pixel to be correctly estimated if the disparity or flow end-point error is <3px or <5% (for scene flow this criterion needs to be fulfilled for both disparity maps and the flow map). We require that all methods use the same parameter set for all test pairs. Our development kit provides details about the data format as well as MATLAB / C++ utility functions for reading and writing disparity maps and flow fields. More details can be found in Object Scene Flow for Autonomous Vehicles (CVPR 2015).

Our evaluation table ranks all methods according to the number of erroneous pixels. All methods providing less than 100 % density have been interpolated using simple background interpolation as explained in the corresponding header file in the development kit. Legend:

  • D1: Percentage of stereo disparity outliers in first frame
  • D2: Percentage of stereo disparity outliers in second frame
  • Fl: Percentage of optical flow outliers
  • SF: Percentage of scene flow outliers (=outliers in either D0, D1 or Fl)
  • bg: Percentage of outliers averaged only over background regions
  • fg: Percentage of outliers averaged only over foreground regions
  • all: Percentage of outliers averaged over all ground truth pixels

Additional information used by the methods
  • Multiview: Method uses more than 2 temporally adjacent images
  • Motion stereo: Method uses epipolar geometry for computing optical flow
  • Additional training data: Use of additional data sources for training (see details)
Evaluation ground truth        Evaluation area

Method Setting Code D1-bg D1-fg D1-all D2-bg D2-fg D2-all Fl-bg Fl-fg Fl-all SF-bg SF-fg SF-all Density Runtime Environment
1 PRSM
This method makes use of multiple (>2) views.
code 3.02 10.52 4.27 5.13 15.11 6.79 5.33 17.02 7.28 6.61 23.60 9.44 99.99 300 s 1 core @ 2.5 Ghz (C/C++)
C. Vogel, K. Schindler and S. Roth: 3D Scene Flow Estimation with a Piecewise Rigid Scene Model. ijcv 2015.
2 OSF+TC
This method makes use of multiple (>2) views.
4.11 9.64 5.03 5.18 15.12 6.84 5.76 16.61 7.57 7.08 22.55 9.65 100.00 50 min 1 core @ 2.5 Ghz (C/C++)
M. Neoral and J. Šochman: Object Scene Flow with Temporal Consistency. 22nd Computer Vision Winter Workshop (CVWW) 2017.
3 SSFAV 3.55 8.94 4.45 4.94 17.36 7.01 5.61 20.80 8.14 7.16 27.91 10.61 100.00 5 min 1 core @ 2.5 Ghz (Matlab + C/C++)
4 OSF code 4.54 12.03 5.79 5.45 19.41 7.77 5.62 22.17 8.37 7.01 28.76 10.63 100.00 50 min 1 core @ 2.5 Ghz (C/C++)
M. Menze and A. Geiger: Object Scene Flow for Autonomous Vehicles. Conference on Computer Vision and Pattern Recognition (CVPR) 2015.
5 SOSF 4.30 9.04 5.09 5.13 17.60 7.20 5.41 23.82 8.47 6.93 30.77 10.90 100.00 55 min 1 core @ 2.5 Ghz (Matlab + C/C++)
6 FSF+MS
This method makes use of the epipolar geometry.
This method makes use of multiple (>2) views.
5.72 11.84 6.74 7.57 21.28 9.85 8.48 29.62 12.00 11.17 37.40 15.54 100.00 2.7 s 4 cores @ 3.5 Ghz (C/C++)
7 CSF 4.57 13.04 5.98 7.92 20.76 10.06 10.40 30.33 13.71 12.21 36.97 16.33 99.99 80 s 1 core @ 2.5 Ghz (C/C++)
Z. Lv, C. Beall, P. Alcantarilla, F. Li, Z. Kira and F. Dellaert: A Continuous Optimization Approach for Efficient and Accurate Scene Flow. European Conf. on Computer Vision (ECCV) 2016.
8 PR-Sceneflow code 4.74 13.74 6.24 11.14 20.47 12.69 11.73 27.73 14.39 13.49 33.72 16.85 100.00 150 s 4 core @ 3.0 Ghz (Matlab + C/C++)
C. Vogel, K. Schindler and S. Roth: Piecewise Rigid Scene Flow. ICCV 2013.
9 SGM+SF 5.15 15.29 6.84 14.10 23.13 15.60 20.91 28.90 22.24 23.09 37.12 25.43 100.00 45 min 16 core @ 3.2 Ghz (C/C++)
H. Hirschmüller: Stereo Processing by Semiglobal Matching and Mutual Information. PAMI 2008.
M. Hornacek, A. Fitzgibbon and C. Rother: SphereFlow: 6 DoF Scene Flow from RGB-D Pairs. CVPR 2014.
10 PCOF-LDOF 6.31 19.24 8.46 19.09 30.54 20.99 14.34 41.30 18.83 25.26 51.55 29.63 100.00 50 s 1 core @ 3.0 Ghz (C/C++)
M. Derome, A. Plyer, M. Sanfourche and G. Le Besnerais: A Prediction-Correction Approach for Real-Time Optical Flow Computation Using Stereo. German Conference on Pattern Recognition 2016.
11 PCOF + ACTF 6.31 19.24 8.46 19.15 36.27 22.00 14.89 62.42 22.80 25.77 69.35 33.02 100.00 0.08 s GPU @ 2.0 Ghz (C/C++)
M. Derome, A. Plyer, M. Sanfourche and G. Le Besnerais: A Prediction-Correction Approach for Real-Time Optical Flow Computation Using Stereo. German Conference on Pattern Recognition 2016.
12 SGM+C+NL code 5.15 15.29 6.84 28.77 25.65 28.25 34.24 45.40 36.10 38.21 53.04 40.68 100.00 4.5 min 1 core @ 2.5 Ghz (C/C++)
H. Hirschmüller: Stereo Processing by Semiglobal Matching and Mutual Information. PAMI 2008.
D. Sun, S. Roth and M. Black: A Quantitative Analysis of Current Practices in Optical Flow Estimation and the Principles Behind Them. IJCV 2013.
13 SGM+LDOF code 5.15 15.29 6.84 29.58 23.48 28.56 40.81 35.42 39.91 43.99 44.79 44.12 100.00 86 s 1 core @ 2.5 Ghz (C/C++)
H. Hirschmüller: Stereo Processing by Semiglobal Matching and Mutual Information. PAMI 2008.
T. Brox and J. Malik: Large Displacement Optical Flow: Descriptor Matching in Variational Motion Estimation. PAMI 2011.
14 DWBSF 19.61 22.69 20.12 35.72 28.15 34.46 40.74 35.53 39.87 46.42 43.99 46.02 100.00 7 min 4 cores @ 3.5 Ghz (C/C++)
C. Richardt, H. Kim, L. Valgaerts and C. Theobalt: Dense Wide-Baseline Scene Flow From Two Handheld Video Cameras. 3DV 2016.
15 GCSF code 11.64 27.11 14.21 32.94 35.77 33.41 47.38 45.08 47.00 52.92 59.11 53.95 100.00 2.4 s 1 core @ 2.5 Ghz (C/C++)
J. Cech, J. Sanchez-Riera and R. Horaud: Scene Flow Estimation by growing Correspondence Seeds. CVPR 2011.
16 VSF code 27.31 21.72 26.38 59.51 44.93 57.08 50.06 47.57 49.64 67.69 64.03 67.08 100.00 125 min 1 core @ 2.5 Ghz (C/C++)
F. Huguet and F. Devernay: A Variational Method for Scene Flow Estimation from Stereo Sequences. ICCV 2007.
Table as LaTeX | Only published Methods



Related Datasets

  • HCI/Bosch Robust Vision Challenge: Optical flow and stereo vision challenge on high resolution imagery recorded at a high frame rate under diverse weather conditions (e.g., sunny, cloudy, rainy). The Robert Bosch AG provides a prize for the best performing method.
  • Image Sequence Analysis Test Site (EISATS): Synthetic image sequences with ground truth information provided by UoA and Daimler AG. Some of the images come with 3D range sensor information.
  • Middlebury Stereo Evaluation: The classic stereo evaluation benchmark, featuring four test images in version 2 of the benchmark, with very accurate ground truth from a structured light system. 38 image pairs are provided in total.
  • Daimler Stereo Dataset: Stereo bad weather highway scenes with partial ground truth for freespace
  • Make3D Range Image Data: Images with small-resolution ground truth used to learn and evaluate depth from single monocular images.
  • Lubor Ladicky's Stereo Dataset: Stereo Images with manually labeled ground truth based on polygonal areas.
  • Middlebury Optical Flow Evaluation: The classic optical flow evaluation benchmark, featuring eight test images, with very accurate ground truth from a shape from UV light pattern system. 24 image pairs are provided in total.

Citation

When using this dataset in your research, we will be happy if you cite us:
@INPROCEEDINGS{Menze2015CVPR,
  author = {Moritz Menze and Andreas Geiger},
  title = {Object Scene Flow for Autonomous Vehicles},
  booktitle = {Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2015}
}
@INPROCEEDINGS{Menze2015ISA,
  author = {Moritz Menze and Christian Heipke and Andreas Geiger},
  title = {Joint 3D Estimation of Vehicles and Scene Flow},
  booktitle = {ISPRS Workshop on Image Sequence Analysis (ISA)},
  year = {2015}
}



eXTReMe Tracker