Dataset


The stereo / flow benchmark consists of 194 training image pairs and 195 test image pairs, saved in loss less png format. Our evaluation server computes the average number of bad pixels for all non-occluded or occluded (=all groundtruth) pixels. We require that all methods use the same parameter set for all test pairs. Our development kit provides details about the data format as well as MATLAB / C++ utility functions for reading and writing disparity maps and flow fields.

Evaluation

Our evaluation table ranks all methods according to the number of non-occluded erroneous pixels at the specified disparity / end-point error threshold. All methods providing less than 100 % density have been interpolated using simple background interpolation as explained in the corresponding header file in the development kit. For each method we show:

  • Out-Noc: Percentage of erroneous pixels in non-occluded areas
  • Out-All: Percentage of erroneous pixels in total
  • Avg-Noc: Average disparity / end-point error in non-occluded areas
  • Avg-All: Average disparity / end-point error in total
  • Density: Percentage of pixels for which ground truth has been provided by the method

Note: Our main ranking is computed at 3 pixels error threshold, evaluating all pixels. For methods which do not provide dense result we use background interpolation to fill in missing values.VERY IMPORTANT NOTE: On 04.11.2013 we have improved the ground truth disparity maps and flow fields leading to slightly improvements for all methods. Please download the stereo/flow dataset with the improved ground truth for training again, if you have downloaded the dataset prior to 04.11.2013. Please consider reporting these new number for all future submissions. The last leaderboards right before the changes can be found here: stereo and flow!

Additional information used by the methods
  • Stereo: Method uses left and right (stereo) images
  • Multiview: Method uses more than 2 temporally adjacent images
  • Motion stereo: Method uses epipolar geometry for computing optical flow

Error threshold        Evaluation area

Optical Flow Evaluation

This table ranks general optical flow methods, performing a full 2D search, as compared to the motion stereo methods below.

Rank Method Setting Code Out-Noc Out-All Avg-Noc Avg-All Density Runtime Environment
1 ECCV_492
This method uses stereo information.
This method makes use of multiple (>2) views.
2.72 % 4.84 % 0.8 px 1.3 px 100.00 % 300 s 1 core @ 2.5 Ghz (Matlab + C/C++)
Anonymous submission
2 ECCV 753
This method uses stereo information.
This method makes use of the epipolar geometry.
2.82 % 5.61 % 0.8 px 1.3 px 100.00 % 35 s 1 core @ 3.5 Ghz (C/C++)
Anonymous submission
3 PR-Sf+E
This method uses stereo information.
3.57 % 7.07 % 0.9 px 1.6 px 100.00 % 200 s 4 cores @ 3.0 Ghz (Matlab + C/C++)
C. Vogel, S. Roth and K. Schindler: Piecewise Rigid Scene Flow. International Conference on Computer Vision (ICCV) 2013.
4 PCBP-Flow
This method makes use of the epipolar geometry.
3.64 % 8.28 % 0.9 px 2.2 px 100.00 % 3 min 4 cores @ 2.5 Ghz (Matlab + C/C++)
K. Yamaguchi, D. McAllester and R. Urtasun: Robust Monocular Epipolar Flow Estimation. CVPR 2013.
5 PR-Sceneflow
This method uses stereo information.
3.76 % 7.39 % 1.2 px 2.8 px 100.00 % 150 sec 4 core @ 3.0 Ghz (Matlab + C/C++)
C. Vogel, S. Roth and K. Schindler: Piecewise Rigid Scene Flow. International Conference on Computer Vision (ICCV) 2013.
6 MotionSLIC
This method makes use of the epipolar geometry.
3.91 % 10.56 % 0.9 px 2.7 px 100.00 % 11 s 1 core @ 3.0 Ghz (C/C++)
K. Yamaguchi, D. McAllester and R. Urtasun: Robust Monocular Epipolar Flow Estimation. CVPR 2013.
7 NLTGV-SC 5.93 % 11.96 % 1.6 px 3.8 px 100.00 % 16 s GPU @ 2.5 Ghz (Matlab + C/C++)
Anonymous submission
8 DDS-DF 6.03 % 13.08 % 1.6 px 4.2 px 100.00 % 1 min 1 core @ 2.5 Ghz (Matlab + C/C++)
Anonymous submission
9 TGV2ADCSIFT 6.20 % 15.15 % 1.5 px 4.5 px 100.00 % 12s GPU @ 2.4 Ghz (C/C++)
J. Braux-Zin, R. Dupont and A. Bartoli: A General Dense Image Matching Framework Combining Direct and Feature-based Costs. International Conference on Computer Vision (ICCV) 2013.
10 AnyFlow 6.37 % 15.80 % 1.5 px 4.3 px 100.00 % 15 s GPU @ 2.5 Ghz (C/C++)
Anonymous submission
11 ECCV-1212 6.52 % 11.03 % 1.5 px 2.8 px 100.00 % 3 min 1 core @ 3.0 Ghz (C/C++)
Anonymous submission
12 Data-Flow 7.11 % 14.57 % 1.9 px 5.5 px 100.00 % 3 min 2 cores @ 2.5 Ghz (Matlab + C/C++)
C. Vogel, S. Roth and K. Schindler: An Evaluation of Data Costs for Optical Flow. German Conference on Pattern Recognition (GCPR) 2013.
13 EpicFlow 7.19 % 16.15 % 1.4 px 3.7 px 100.00 % 15 s 1 core @ 3.6 Ghz (C/C++)
Anonymous submission
14 DeepFlow 7.22 % 17.79 % 1.5 px 5.8 px 100.00 % 17 s 1 core @ 3.6Ghz (Python + C/C++)
P. Weinzaepfel, J. Revaud, Z. Harchaoui and C. Schmid: DeepFlow: Large displacement optical flow with deep matching. IEEE Intenational Conference on Computer Vision (ICCV) 2013.
15 TVL1-HOG 7.91 % 18.90 % 2.0 px 6.1 px 100.00 % 180 s 2 cores @ 3.0 Ghz (Matlab)
H. Rashwan, M. Mohamed, M. Garcia, B. Mertsching and D. Puig: Illumination Robust Optical Flow Model Based on Histogram of Oriented Gradients. German Conference on Pattern Recognition 2013 .
16 MLDP-OF 8.67 % 18.78 % 2.4 px 6.7 px 100.00 % 160 s 2 cores @ 2.5 Ghz (Matlab)
Anonymous submission
17 DescFlow 8.76 % 19.45 % 2.1 px 5.7 px 100.00 % 9.0 s GPU @ 2.5 Ghz (C/C++)
Anonymous submission
18 CRTflow 9.43 % 18.72 % 2.7 px 6.5 px 100.00 % 18 s GPU @ 1.0 Ghz (C/C++)
O. Demetz, D. Hafner and J. Weickert: The Complete Rank Transform: A Tool for Accurate and Morphologically Invariant Matching of Structure. Proc.~British Machine Vision Conference 2013 (BMVC) 2013.
19 C++ code 10.04 % 20.26 % 2.6 px 7.1 px 100.00 % 8.5 min 1 core @ 3.0 Ghz (Matlab)
D. Sun, S. Roth and M. Black: A Quantitative Analysis of Current Practices in Optical Flow Estimation and The Principles Behind Them. 2014.
20 C+NL code 10.49 % 20.64 % 2.8 px 7.2 px 100.00 % 14.8 min 1 core @ 3.0 Ghz (Matlab)
D. Sun, S. Roth and M. Black: A Quantitative Analysis of Current Practices in Optical Flow Estimation and The Principles Behind Them. 2014.
21 IVANN 10.68 % 21.09 % 2.7 px 7.4 px 100.00 % 1073 s 1 core @ 2.5 Ghz (Matlab)
Anonymous submission
22 fSGM 10.74 % 22.66 % 3.2 px 12.2 px 100.00 % 60 s 1 core @ 2.4 Ghz (C/C++)
S. Hermann and R. Klette: Hierarchical Scan Line Dynamic Programming for Optical Flow using Semi-Global Matching. ACCV Workshops 2012.
23 TGV2CENSUS code 11.03 % 18.37 % 2.9 px 6.6 px 100.00 % 4 s GPU+CPU @ 3.0 Ghz (Matlab + C/C++)
M. Werlberger: Convex Approaches for High Performance Video Processing. 2012.
R. Ranftl, S. Gehrig, T. Pock and H. Bischof: Pushing the Limits of Stereo Using Variational Stereo Estimation. IV 2012.
24 C+NL-fast code 12.36 % 22.28 % 3.2 px 7.9 px 100.00 % 2.9 min 1 core @ 3.0 Ghz (Matlab)
D. Sun, S. Roth and M. Black: A Quantitative Analysis of Current Practices in Optical Flow Estimation and The Principles Behind Them. 2014.
25 TriFlowFused 12.58 % 22.91 % 2.4 px 5.8 px 100.00 % 350 s 4 cores @ 3.0 Ghz (Matlab + C/C++)
Anonymous submission
26 EPPM code 12.75 % 23.55 % 2.5 px 9.2 px 100.00 % 0.25 s GPU @ 1.0 Ghz (C/C++)
L. Bao, Q. Yang and H. Jin: Fast Edge-Preserving PatchMatch for Large Displacement Optical Flow. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2014.
27 HS code 14.75 % 24.11 % 4.0 px 9.0 px 100.00 % 2.6 min 1 core @ 3.0 Ghz (Matlab)
D. Sun, S. Roth and M. Black: A Quantitative Analysis of Current Practices in Optical Flow Estimation and The Principles Behind Them. 2014.
28 GC-BM-Bino
This method uses stereo information.
This method makes use of the epipolar geometry.
18.83 % 29.30 % 5.0 px 12.1 px 83.73 % 1.3 s 2 cores @ 2.5 Ghz (C/C++)
B. Kitt and H. Lategahn: Trinocular Optical Flow Estimation for Intelligent Vehicle Applications. ITSC 2012.
29 IQFlow 18.84 % 28.25 % 3.6 px 8.8 px 100.00 % 60 s 4 cores @ 3.5 Ghz (C/C++)
Anonymous submission
30 C+NL-M 19.19 % 26.36 % 7.4 px 14.5 px 100.00 % 5 min 2 cores @ 2.5 Ghz (Matlab)
Anonymous submission
31 eFolki 19.31 % 28.79 % 5.2 px 10.9 px 100.00 % 0.026 s GPU @ 700 Mhz (C/C++)
Anonymous submission
32 GC-BM-Mono
This method makes use of the epipolar geometry.
19.38 % 29.80 % 5.0 px 12.1 px 84.33 % 1.3 s 2 cores @ 2.5 Ghz (C/C++)
B. Kitt and H. Lategahn: Trinocular Optical Flow Estimation for Intelligent Vehicle Applications. ITSC 2012.
33 RSRS-Flow 20.78 % 29.75 % 6.2 px 12.1 px 100.00 % 4 min 1 core @ 2.5 Ghz (Matlab)
P. Ghosh and B. Manjunath: Robust Simultaneous Registration and Segmentation with Sparse Error Reconstruction. PAMI 2012.
34 ALD 21.37 % 30.71 % 10.9 px 16.0 px 100.00 % 110 s 1 core @ 2.5 Ghz (C/C++)
M. Stoll, S. Volz and A. Bruhn: Adaptive Integration of Feature Matches into Variational Optical Flow Methods. ACCV 2012.
35 LDOF code 21.93 % 31.39 % 5.6 px 12.4 px 100.00 % 1 min 1 core @ 2.5 Ghz (C/C++)
T. Brox and J. Malik: Large Displacement Optical Flow: Descriptor Matching in Variational Motion Estimation. PAMI 2011.
36 HMM 24.78 % 34.19 % 7.2 px 15.1 px 100.00 % 10 min 1 core @ 2.5 Ghz (C/C++)
Anonymous submission
37 PCA-Flow 27.65 % 36.95 % 3.6 px 7.3 px 100.00 % 0.18 s 1 core @ 2.0 Ghz (Python)
Anonymous submission
38 DB-TV-L1 code 30.87 % 39.25 % 7.9 px 14.6 px 100.00 % 16 s 1 core @ 2.5 Ghz (Matlab)
C. Zach, T. Pock and H. Bischof: A Duality Based Approach for Realtime TV- L1 Optical Flow. DAGM 2007.
39 GCSF
This method uses stereo information.
33.17 % 41.71 % 7.0 px 15.3 px 48.27 % 2.4 s 1 core @ 2.5 Ghz (C/C++)
J. Cech, J. Sanchez-Riera and R. Horaud: Scene Flow Estimation by Growing Correspondence Seeds. CVPR 2011.
40 HAOF code 35.87 % 43.46 % 11.1 px 18.3 px 100.00 % 16.2 s 1 core @ 2.5 Ghz (C/C++)
T. Brox, A. Bruhn, N. Papenberg and J. Weickert: High accuracy optical flow estimation based on a theory for warping. ECCV 2004.
41 2-Bit code 36.13 % 43.99 % 10.1 px 20.3 px 100.00 % 90 min 1 core @ 2.5 Ghz (C/C++)
R. Xu and D. Taubman: Robust Dense Block-Based Motion Estimation Using a Two-Bit Transform on a Laplacian Pyramid. 20th Proc. IEEE Int. Conf. Image Proc. 2013 2013.
42 BERLOF 37.66 % 45.27 % 8.5 px 16.2 px 15.26 % 0.231 s GPU @ 700 Mhz (C/C++) GeForce GTX 680
T. Senst, J. Geistert, I. Keller and T. Sikora: Robust Local Optical Flow Estimation using Bilinear Equations for Sparse Motion Estimation. 20th IEEE International Conference on Image Processing 2013.
43 RLOF code 38.60 % 46.13 % 8.7 px 16.5 px 14.76 % 0.488 s GPU @ 700 Mhz (C/C++) GeForce GTX 680
T. Senst, V. Eiselein and T. Sikora: Robust Local Optical Flow for Feature Tracking. TCSVT 2012.
44 SpaGloM 41.91 % 48.59 % 8.6 px 15.3 px 17.14 % 50 s 1 core @ 2.5 Ghz (C/C++)
Anonymous submission
45 PolyExpand 47.59 % 54.00 % 17.3 px 25.3 px 100.00 % 1 s 1 core @ 2.5 Ghz (C/C++)
G. Farneback: Two-Frame Motion Estimation Based on Polynomial Expansion. SCIA 2003.
46 OCV-BM code 63.50 % 68.19 % 24.4 px 33.3 px 100.00 % 1.5 min 1 core @ 2.5 Ghz (C/C++)
G. Bradski: The OpenCV Library. Dr. Dobb's Journal of Software Tools 2000.
47 Pyramid-LK code 65.81 % 70.16 % 21.8 px 33.2 px 99.90 % 1.5 min 1 core @ 2.5 Ghz (Matlab)
J. Bouguet: Pyramidal implementation of the Lucas Kanade feature tracker. Intel 2000.
48 MEDIAN 79.37 % 82.46 % 16.0 px 24.0 px 99.94 % 0.01 s 1 core @ 2.5 Ghz (C/C++)
49 AVERAGE 81.27 % 84.06 % 16.3 px 24.7 px 99.94 % 0.01 s 1 core @ 2.5 Ghz (C/C++)
This table as LaTeX

The settings column describes additional assumptions made / information used by the methods:

  • ms = motion stereo: Usage of the epipolar geometry to restrict the search problem to 1D

Related Datasets

  • HCI/Bosch Robust Vision Challenge: Optical flow and stereo vision challenge on high resolution imagery recorded at a high frame rate under diverse weather conditions (e.g., sunny, cloudy, rainy). The Robert Bosch AG provides a prize for the best performing method.
  • Image Sequence Analysis Test Site (EISATS): Synthetic image sequences with ground truth information provided by UoA and Daimler AG. Some of the images come with 3D range sensor information.
  • Middlebury Optical Flow Evaluation: The classic optical flow evaluation benchmark, featuring eight test images, with very accurate ground truth from a shape from UV light pattern system. 24 image pairs are provided in total.



eXTReMe Tracker