Dataset


The stereo / flow benchmark consists of 194 training image pairs and 195 test image pairs, saved in loss less png format. Our evaluation server computes the average number of bad pixels for all non-occluded or occluded (=all groundtruth) pixels. We require that all methods use the same parameter set for all test pairs. Our development kit provides details about the data format as well as MATLAB / C++ utility functions for reading and writing disparity maps and flow fields.

Evaluation

Our evaluation table ranks all methods according to the number of non-occluded erroneous pixels at the specified disparity / end-point error threshold. All methods providing less than 100 % density have been interpolated using simple background interpolation as explained in the corresponding header file in the development kit. For each method we show:

  • Out-Noc: Percentage of erroneous pixels in non-occluded areas
  • Out-All: Percentage of erroneous pixels in total
  • Avg-Noc: Average disparity / end-point error in non-occluded areas
  • Avg-All: Average disparity / end-point error in total
  • Density: Percentage of pixels for which ground truth has been provided by the method

Note: Our main ranking is computed at 3 pixels error threshold, evaluating all pixels. For methods which do not provide dense result we use background interpolation to fill in missing values.

  • Stereo: Method uses left and right (stereo) images
  • Flow: Method uses optical flow (2 temporally adjacent images)
  • Multiview: Method uses more than 2 temporally adjacent images
  • Laser Points: Method uses point clouds from Velodyne laser scanner
  • GPS: Method uses GPS information
  • Motion stereo: Method uses epipolar geometry for computing optical flow

Error threshold        Evaluation area

Stereo Evaluation


Rank Method Setting Code Out-Noc Out-All Avg-Noc Avg-All Density Runtime Environment
1 SceneFlow
This method uses stereo information.
This method uses optical flow information.
This method makes use of the epipolar geometry.
3.03 % 3.90 % 0.8 px 1.0 px 100.00 % 6 min 4 cores @ 3.0 Ghz (Matlab + C/C++)
Anonymous submission
2 PCBP-SS
This method uses stereo information.
3.49 % 4.79 % 0.8 px 1.0 px 100.00 % 5 min 4 cores @ 2.5 Ghz (Matlab + C/C++)
K. Yamaguchi, D. McAllester and R. Urtasun: Robust Monocular Epipolar Flow Estimation. CVPR 2013.
3 gtRF-SS
This method uses stereo information.
3.91 % 4.68 % 0.9 px 1.0 px 100.00 % 1 min 1 core @ 2.5 Ghz (Matlab + C/C++)
Anonymous submission
4 StereoSLIC
This method uses stereo information.
3.99 % 5.17 % 0.9 px 1.0 px 99.89 % 2.3 s 1 core @ 3.0 Ghz (C/C++)
K. Yamaguchi, D. McAllester and R. Urtasun: Robust Monocular Epipolar Flow Estimation. CVPR 2013.
5 PR-Sf+E
This method uses stereo information.
This method uses optical flow information.
4.09 % 4.95 % 0.9 px 1.0 px 100.00 % 200 s 4 cores @ 3.0 Ghz (Matlab + C/C++)
C. Vogel, S. Roth and K. Schindler: Piecewise Rigid Scene Flow. International Conference on Computer Vision (ICCV) 2013.
6 PCBP
This method uses stereo information.
4.13 % 5.45 % 0.9 px 1.2 px 100.00 % 5 min 4 cores @ 2.5 Ghz (Matlab + C/C++)
K. Yamaguchi, T. Hazan, D. McAllester and R. Urtasun: Continuous Markov Random Fields for Robust Stereo Estimation. ECCV 2012.
7 PR-Sceneflow
This method uses stereo information.
This method uses optical flow information.
4.46 % 5.32 % 1.0 px 1.1 px 100.00 % 150 sec 4 core @ 3.0 Ghz (Matlab - C/C++)
C. Vogel, S. Roth and K. Schindler: Piecewise Rigid Scene Flow. International Conference on Computer Vision (ICCV) 2013.
8 wSGM
This method uses stereo information.
5.03 % 6.24 % 1.3 px 1.6 px 97.03 % 6s 1 core @ 3.5 Ghz (C/C++)
T. Robert Spangenberg and R. Rojas: Weighted Semi-Global Matching and Center-Symmetric Census Transform for Robust Driver Assistance. CAIP 2013.
9 ATGV
This method uses stereo information.
5.05 % 6.91 % 1.0 px 1.6 px 100.00 % 6 min >8 cores @ 3.0 Ghz (Matlab + C/C++)
R. Ranftl, T. Pock and H. Bischof: Minimizing TGV-based Variational Models with Non-Convex Data terms. ICSSVM 2013.
10 iSGM
This method uses stereo information.
5.16 % 7.19 % 1.2 px 2.1 px 94.70 % 8 s 2 cores @ 2.5 Ghz (C/C++)
S. Hermann and R. Klette: Iterative Semi-Global Matching for Robust Driver Assistance Systems. ACCV 2012.
11 OCV-SGBM2
This method uses stereo information.
5.42 % 6.54 % 1.0 px 1.2 px 100.00 % 2 s 1 core @ 2.5 Ghz (C/C++)
Anonymous submission
12 ALTGV
This method uses stereo information.
5.48 % 6.60 % 1.1 px 1.2 px 100.00 % 20 s GPU @ 2.5 Ghz (C/C++)
Anonymous submission
13 AABM
This method uses stereo information.
5.50 % 6.60 % 1.1 px 1.3 px 100.00 % 0.43 s 1 core @ 3.0 Ghz (C/C++)
N. Einecke and J. Eggert: Appearance-Aligned Block-Matching Stereo. IV 2013.
14 SGM
This method uses stereo information.
5.83 % 7.08 % 1.2 px 1.3 px 85.80 % 3.7 s 1 core @ 3.0 Ghz (C/C++)
H. Hirschmueller: Stereo Processing by Semi-Global Matching and Mutual Information. PAMI 2008.
15 TGV2ADC
This method uses stereo information.
6.02 % 6.94 % 1.1 px 1.2 px 99.99 % 8 s GPU @ 2.5 Ghz (C/C++)
Anonymous submission
16 mSGM-LDE
This method uses stereo information.
6.11 % 8.32 % 1.4 px 2.4 px 100.00 % 55 s 2 cores @ 2.5 Ghz (C/C++)
Anonymous submission
17 CD
This method uses stereo information.
6.17 % 7.49 % 1.2 px 1.4 px 100.00 % 5 s 1 core @ 2.5 Ghz (C/C++)
Anonymous submission
18 SNCC
This method uses stereo information.
6.27 % 7.33 % 1.4 px 1.5 px 100.00 % 0.27 s 1 core @ 3.0 Ghz (C/C++)
N. Einecke and J. Eggert: A Two-Stage Correlation Method for Stereoscopic Depth Estimation. DICTA 2010.
19 ITGV
This method uses stereo information.
6.31 % 7.40 % 1.3 px 1.5 px 100.00 % 7 s 1 core @ 3.0 Ghz (Matlab + C/C++)
R. Ranftl, S. Gehrig, T. Pock and H. Bischof: Pushing the Limits of Stereo Using Variational Stereo Estimation. IV 2012.
20 RWR
This method uses stereo information.
6.36 % 7.51 % 1.2 px 1.4 px 100.00 % 1 min 1 core @ 2.5 Ghz (C/C++)
Anonymous submission
21 RWR+Gradient
This method uses stereo information.
6.48 % 7.56 % 1.4 px 1.6 px 99.99 % 5 s 1 core @ 2.5 Ghz (C/C++)
Anonymous submission
22 LDE
This method uses stereo information.
6.81 % 8.92 % 1.8 px 2.5 px 100.00 % 14 s 2 cores @ 2.5 Ghz (C/C++)
Anonymous submission
23 BSSM
This method uses stereo information.
7.50 % 8.89 % 1.4 px 1.6 px 94.87 % 20.7 s 1 core @ 3.5 Ghz (C/C++)
Anonymous submission
24 OCV-SGBM
This method uses stereo information.
code 7.64 % 9.13 % 1.8 px 2.0 px 86.50 % 1.1 s 1 core @ 2.5 Ghz (C/C++)
H. Hirschmueller: Stereo processing by semiglobal matching and mutual information. PAMI 2008.
25 TV-WL1+ELAS
This method uses stereo information.
8.08 % 8.41 % 1.5 px 1.6 px 97.84 % 1 min 1 core @ 2.5 Ghz (Matlab + C/C++)
Anonymous submission
26 ELAS
This method uses stereo information.
code 8.24 % 9.95 % 1.4 px 1.6 px 94.55 % 0.3 s 1 core @ 2.5 Ghz (C/C++)
A. Geiger, M. Roser and R. Urtasun: Efficient Large-Scale Stereo Matching. ACCV 2010.
27 linBP
This method uses stereo information.
8.66 % 10.81 % 1.7 px 2.7 px 99.89 % 1.6 min 1 core @ 3.0 Ghz (C/C++)
W. Khan, V. Suaste, D. Caudillo and R. Klette: Belief Propagation Stereo Matching Compared to iSGM on Binocular or Trinocular Video Data. IV 2013.
28 SM_GPTM
This method uses stereo information.
9.86 % 11.45 % 2.1 px 2.6 px 100.00 % 6.5 s 2 cores @ 2.5 Ghz (C/C++)
C. Cigla and A. Alatan: An Improved Stereo Matching Algorithm with Ground Plane and Temporal Smoothness Constraints. ECCV Workshops 2012.
29 LAMC-DSΜ
This method uses stereo information.
9.90 % 11.56 % 2.1 px 2.7 px 99.96 % 10.8 min 2 cores @ 2.5 Ghz (Matlab)
C. Stentoumis, L. Grammatikopoulos, I. Kalisperakis, E. Petsa and G. Karras: A local adaptive approach for dense stereo matching in architectural scene reconstruction. ISPRS 2013.
30 MS-DSI
This method uses stereo information.
10.68 % 12.11 % 1.9 px 2.2 px 100.00 % 10.3 s >8 cores @ 2.5 Ghz (C/C++)
Anonymous submission
31 SDM
This method uses stereo information.
code 10.98 % 12.19 % 2.0 px 2.3 px 63.58 % 1 min 1 core @ 2.5 Ghz (C/C++)
J. Kostkova: Stratified dense matching for stereopsis in complex scenes. BMVC 2003.
32 BSM
This method uses stereo information.
code 11.85 % 13.54 % 2.2 px 2.8 px 97.02 % 2.5 min 1 core @ 3.0 Ghz (C/C++)
K. Zhang, J. Li, Y. Li, W. Hu, L. Sun and S. Yang: Binary stereo matching. Pattern Recognition (ICPR), 2012 21st International Conference on 2012.
33 GCSF
This method uses stereo information.
This method uses optical flow information.
12.06 % 13.26 % 1.9 px 2.1 px 60.77 % 2.4 s 1 core @ 2.5 Ghz (C/C++)
J. Cech, J. Sanchez-Riera and R. Horaud: Scene Flow Estimation by Growing Correspondence Seeds. CVPR 2011.
34 OCV-BM-post
This method uses stereo information.
code 12.20 % 13.68 % 2.1 px 2.3 px 47.11 % 0.1 s 1 core @ 2.5 Ghz (C/C++)
G. Bradski: The OpenCV Library. Dr. Dobb's Journal of Software Tools 2000.
35 GCS
This method uses stereo information.
code 13.37 % 14.54 % 2.1 px 2.3 px 51.06 % 2.2 s 1 core @ 2.5 Ghz (C/C++)
J. Cech and R. Sara: Efficient Sampling of Disparity Space for Fast And Accurate Matching. BenCOS 2007.
36 MPA-1
This method uses stereo information.
19.04 % 20.96 % 4.9 px 6.3 px 100.00 % 4 min 1 core @ 2.5 Ghz (Matlab)
Anonymous submission
37 CostFilter
This method uses stereo information.
code 19.96 % 21.05 % 5.0 px 5.4 px 100.00 % 4 min 1 core @ 2.5 Ghz (Matlab)
C. Rhemann, A. Hosni, M. Bleyer, C. Rother and M. Gelautz: Fast Cost-Volume Filtering for Visual Correspondence and Beyond. CVPR 2011.
38 MPHVA
This method uses stereo information.
23.78 % 25.58 % 6.8 px 8.0 px 100.00 % 300 s 1 core @ 2.5 Ghz (Matlab)
Anonymous submission
39 OCV-BM
This method uses stereo information.
code 25.39 % 26.72 % 7.6 px 7.9 px 55.84 % 0.1 s 1 core @ 2.5 Ghz (C/C++)
G. Bradski: The OpenCV Library. Dr. Dobb's Journal of Software Tools 2000.
40 GC+occ
This method uses stereo information.
code 33.50 % 34.74 % 8.6 px 9.2 px 87.57 % 6 min 1 core @ 2.5 Ghz (C/C++)
V. Kolmogorov and R. Zabih: Computing Visual Correspondence with Occlusions using Graph Cuts. ICCV 2001.
41 MEDIAN
This method uses stereo information.
52.75 % 53.81 % 7.7 px 8.2 px 99.95 % 0.01 s 1 core @ 2.5 Ghz (C/C++)
42 AVERAGE
This method uses stereo information.
61.87 % 62.72 % 8.1 px 8.6 px 99.95 % 0.01 s 1 core @ 2.5 Ghz (C/C++)
This table as LaTeX

Related Datasets

  • HCI/Bosch Robust Vision Challenge: Optical flow and stereo vision challenge on high resolution imagery recorded at a high frame rate under diverse weather conditions (e.g., sunny, cloudy, rainy). The Robert Bosch AG provides a prize for the best performing method.
  • Image Sequence Analysis Test Site (EISATS): Synthetic image sequences with ground truth information provided by UoA and Daimler AG. Some of the images come with 3D range sensor information.
  • Middlebury Stereo Evaluation: The classic stereo evaluation benchmark, featuring four test images in version 2 of the benchmark, with very accurate ground truth from a structured light system. 38 image pairs are provided in total.
  • Daimler Stereo Dataset: Stereo bad weather highway scenes with partial ground truth for freespace
  • Make3D Range Image Data: Images with small-resolution ground truth used to learn and evaluate depth from single monocular images.
  • Lubor Ladicky's Stereo Dataset: Stereo Images with manually labeled ground truth based on polygonal areas.



eXTReMe Tracker