Visual Odometry / SLAM Evaluation 2012


The odometry benchmark consists of 22 stereo sequences, saved in loss less png format: We provide 11 sequences (00-10) with ground truth trajectories for training and 11 sequences (11-21) without ground truth for evaluation. For this benchmark you may provide results using monocular or stereo visual odometry, laser-based SLAM or algorithms that combine visual and LIDAR information. The only restriction we impose is that your method is fully automatic (e.g., no manual loop-closure tagging is allowed) and that the same parameter set is used for all sequences. A development kit provides details about the data format.

From all test sequences, our evaluation computes translational and rotational errors for all possible subsequences of length (100,...,800) meters. The evaluation table below ranks methods according to the average of those values, where errors are measured in percent (for translation) and in degrees per meter (for rotation). A more detailed comparison for different trajectory lengths and driving speeds can be found in the plots underneath. Note: On 03.10.2013 we have changed the evaluated sequence lengths from (5,10,50,100,...,400) to (100,200,...,800) due to the fact that the GPS/OXTS ground truth error for very small sub-sequences was large and hence biased the evaluation results. Now the averages below take into account longer sequences and provide a better indication of the true performance. Please consider reporting these number for all future submissions. The last leaderboard right before the changes can be found here!

Additional information used by the methods
  • Stereo: Method uses left and right (stereo) images
  • Laser Points: Method uses point clouds from Velodyne laser scanner
  • Loop Closure Detection: This method is a SLAM method that detects loop closures
  • Additional training data: Use of additional data sources for training (see details)
Method Setting Code Translation Rotation Runtime Environment
1 V-LOAM
This method makes use of Velodyne laser scans.
0.68 % 0.0016 [deg/m] 0.1 s 2 cores @ 2.5 Ghz (C/C++)
J. Zhang and S. Singh: Visual-lidar Odometry and Mapping: Low drift, Robust, and Fast. IEEE International Conference on Robotics and Automation(ICRA) 2015.
2 LOAM
This method makes use of Velodyne laser scans.
0.70 % 0.0017 [deg/m] 0.1 s 2 cores @ 2.5 Ghz (C/C++)
J. Zhang and S. Singh: LOAM: Lidar Odometry and Mapping in Real-time. Robotics: Science and Systems Conference (RSS) 2014.
3 SOFT2
This method uses stereo information.
0.81 % 0.0022 [deg/m] 0.1 s 2 cores @ 2.5 Ghz (C/C++)
4 GDVO
This method uses stereo information.
0.86 % 0.0031 [deg/m] 0.09 s 1 core @ >3.5 Ghz (C/C++)
5 HypERROCC
This method uses stereo information.
0.88 % 0.0027 [deg/m] 0.25 s 2 cores @ 2.0 Ghz (C/C++)
6 SOFT
This method uses stereo information.
0.88 % 0.0022 [deg/m] 0.1 s 2 cores @ 2.5 Ghz (C/C++)
I. Cvišić and I. Petrović: Stereo odometry based on careful feature selection and tracking. European Conference on Mobile Robots (ECMR) 2015.
7 RotRocc
This method uses stereo information.
0.88 % 0.0025 [deg/m] 0.3 s 2 cores @ 2.0 Ghz (C/C++)
M. Buczko and V. Willert: Flow-Decoupled Normalized Reprojection Error for Visual Odometry. 19th IEEE Intelligent Transportation Systems Conference (ITSC) 2016.
8 EDVO
This method uses stereo information.
0.89 % 0.0030 [deg/m] 0.1 s 1 core @ 2.5 Ghz (C/C++)
9 svo2
This method uses stereo information.
0.94 % 0.0021 [deg/m] 0.2 s 1 core @ 2.5 Ghz (C/C++)
10 ROCC
This method uses stereo information.
0.98 % 0.0028 [deg/m] 0.3 s 2 cores @ 2.0 Ghz (C/C++)
M. Buczko and V. Willert: How to Distinguish Inliers from Outliers in Visual Odometry for High-speed Automotive Applications. IEEE Intelligent Vehicles Symposium (IV) 2016.
11 cv4xv1-sc
This method uses stereo information.
1.09 % 0.0029 [deg/m] 0.145 s GPU @ 3.5 Ghz (C/C++)
M. Persson, T. Piccini, R. Mester and M. Felsberg: Robust Stereo Visual Odometry from Monocular Techniques. IEEE Intelligent Vehicles Symposium 2015.
12 MonoROCC
This method uses stereo information.
1.11 % 0.0028 [deg/m] 1 s 2 cores @ 2.0 Ghz (C/C++)
13 DEMO
This method makes use of Velodyne laser scans.
1.14 % 0.0049 [deg/m] 0.1 s 2 cores @ 2.5 Ghz (C/C++)
J. Zhang, M. Kaess and S. Singh: Real-time Depth Enhanced Monocular Odometry. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2014.
14 ORB-SLAM2
This method uses stereo information.
code 1.15 % 0.0027 [deg/m] 0.06 s 2 cores @ >3.5 Ghz (C/C++)
15 svo
This method uses stereo information.
1.16 % 0.0030 [deg/m] 0.1 s 2 core @ 2.5 Ghz (C/C++)
16 NOTF
This method uses stereo information.
1.17 % 0.0035 [deg/m] 0.45 s 1 core @ 3.0 Ghz (C/C++)
J. Deigmoeller and J. Eggert: Stereo Visual Odometry without Temporal Filtering. German Conference on Pattern Recognition (GCPR) 2016.
17 S-PTAM
This method uses stereo information.
code 1.19 % 0.0025 [deg/m] 0.03 s 4 cores @ 3.0 Ghz (C/C++)
T. Pire, T. Fischer, J. Civera, P. Crist\'{o}foris and J. Jacobo-Berlles: Stereo parallel tracking and mapping for robot localization. IROS 2015.
18 S-LSD-SLAM
This method uses stereo information.
code 1.20 % 0.0033 [deg/m] 0.07 s 1 core @ 3.5 Ghz (C/C++)
J. Engel, J. St\"uckler and D. Cremers: Large-Scale Direct SLAM with Stereo Cameras. Int.~Conf.~on Intelligent Robot Systems (IROS) 2015.
19 VoBa
This method uses stereo information.
1.22 % 0.0029 [deg/m] 0.1 s 1 core @ 2.0 Ghz (C/C++)
J. Tardif, M. George, M. Laverne, A. Kelly and A. Stentz: A new approach to vision-aided inertial navigation. 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, October 18-22, 2010, Taipei, Taiwan 2010.
20 SLUP
This method uses stereo information.
1.25 % 0.0041 [deg/m] 0.17 s 4 cores @ 3.3 Ghz (C/C++)
21 FRVO
This method uses stereo information.
1.26 % 0.0038 [deg/m] 0.03 s 1 core @ 3.5 Ghz (C/C++)
22 MFI
This method uses stereo information.
1.30 % 0.0030 [deg/m] 0.1 s 1 core @ 2.2 Ghz (C/C++)
H. Badino, A. Yamamoto and T. Kanade: Visual Odometry by Multi-frame Feature Integration. First International Workshop on Computer Vision for Autonomous Driving at ICCV 2013.
23 TLBBA
This method uses stereo information.
1.36 % 0.0038 [deg/m] 0.1 s 1 Core @2.8GHz (C/C++)
W. Lu, Z. Xiang and J. Liu: High-performance visual odometry with two- stage local binocular BA and GPU. Intelligent Vehicles Symposium (IV), 2013 IEEE 2013.
24 2FO-CC
This method uses stereo information.
code 1.37 % 0.0035 [deg/m] 0.1 s 1 core @ 3.0 Ghz (C/C++)
I. Krešo and S. Šegvić: Improving the Egomotion Estimation by Correcting the Calibration Bias. VISAPP 2015.
25 StereoSFM
This method uses stereo information.
code 1.51 % 0.0042 [deg/m] 0.02 s 2 cores @ 2.5 Ghz (C/C++)
H. Badino and T. Kanade: A Head-Wearable Short-Baseline Stereo System for the Simultaneous Estimation of Structure and Motion. IAPR Conference on Machine Vision Application 2011.
26 M 1.54 % 0.0032 [deg/m] 0.02 s 1 core @ 2.5 Ghz (C/C++)
27 SSLAM
This method uses stereo information.
code 1.57 % 0.0044 [deg/m] 0.5 s 8 cores @ 3.5 Ghz (C/C++)
F. Bellavia, M. Fanfani, F. Pazzaglia and C. Colombo: Robust Selective Stereo SLAM without Loop Closure and Bundle Adjustment. ICIAP 2013 2013.
F. Bellavia, M. Fanfani and C. Colombo: Selective visual odometry for accurate AUV localization. Autonomous Robots 2015.
M. Fanfani, F. Bellavia and C. Colombo: Accurate Keyframe Selection and Keypoint Tracking for Robust Visual Odometry. Machine Vision and Applications 2016.
28 eVO
This method uses stereo information.
1.76 % 0.0036 [deg/m] 0.05 s 2 cores @ 2.0 Ghz (C/C++)
M. Sanfourche, V. Vittori and G. Besnerais: eVO: A realtime embedded stereo odometry for MAV applications. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2013.
29 Stereo DWO
This method uses stereo information.
code 1.76 % 0.0026 [deg/m] 0.1 s 4 cores @ 2.5 Ghz (C/C++)
J. Huai, C. Toth and D. Grejner-Brzezinska: Stereo-inertial odometry using nonlinear optimization. Proceedings of the 27th International Technical Meeting of The Satellite Division of the Institute of Navigation (ION GNSS+ 2015) 2015.
30 D6DVO
This method uses stereo information.
2.04 % 0.0051 [deg/m] 0.03 s 1 core @ 2.5 Ghz (C/C++)
A. Comport, E. Malis and P. Rives: Accurate Quadrifocal Tracking for Robust 3D Visual Odometry. ICRA 2007.
M. Meilland, A. Comport and P. Rives: Dense visual mapping of large scale environments for real-time localisation. ICRA 2011.
31 MB-ICP
This method makes use of Velodyne laser scans.
2.06 % 0.0098 [deg/m] 1 s 1 core @ 2.5 Ghz (C/C++)
32 SSLAM-HR
This method uses stereo information.
code 2.14 % 0.0059 [deg/m] 0.5 s 8 cores @ 3.5 Ghz (C/C++)
F. Bellavia, M. Fanfani, F. Pazzaglia and C. Colombo: Robust Selective Stereo SLAM without Loop Closure and Bundle Adjustment. ICIAP 2013 2013.
F. Bellavia, M. Fanfani and C. Colombo: Selective visual odometry for accurate AUV localization. Autonomous Robots 2015.
M. Fanfani, F. Bellavia and C. Colombo: Accurate Keyframe Selection and Keypoint Tracking for Robust Visual Odometry. Machine Vision and Applications 2016.
33 FTMVO 2.24 % 0.0049 [deg/m] 0.11 s 1 core @ 2.5 Ghz (C/C++)
H. Mirabdollah and B. Mertsching: Fast Techniques for Monocular Visual Odometry . Proceeding of 37th German Conference on Pattern Recognition (GCPR) 2015 .
34 MEVO
This method uses stereo information.
2.40 % 0.0060 [deg/m] 1 s 1 core @ 3.0 Ghz (C/C++)
35 VISO2-S
This method uses stereo information.
code 2.44 % 0.0114 [deg/m] 0.05 s 1 core @ 2.5 Ghz (C/C++)
A. Geiger, J. Ziegler and C. Stiller: StereoScan: Dense 3d Reconstruction in Real-time. IV 2011.
36 MLM-SFM 2.54 % 0.0057 [deg/m] 0.03 s 5 cores @ 2.5 Ghz (C/C++)
S. Song and M. Chandraker: Robust Scale Estimation in Real-Time Monocular SFM for Autonomous Driving. CVPR 2014.
S. Song, M. Chandraker and C. Guest: Parallel, Real-time Monocular Visual Odometry. ICRA 2013.
37 GT_VO3pt
This method uses stereo information.
2.54 % 0.0078 [deg/m] 1.26 s 1 core @ 2.5 Ghz (C/C++)
C. Beall, B. Lawrence, V. Ila and F. Dellaert: 3D reconstruction of underwater structures. IROS 2010.
38 BoofCV-SQ3
This method uses stereo information.
code 2.54 % 0.0073 [deg/m] 0.14 s 1 core @ 2.5 Ghz (Java)
39 RMCPE+GP 2.55 % 0.0086 [deg/m] 0.39 s 1 core @ 2.5 Ghz (C/C++)
M. Mirabdollah and B. Mertsching: On the Second Order Statistics of Essential Matrix Elements. Proceeding of 36th German Conference on Pattern Recognition 2014.
40 SuMO
This method makes use of Velodyne laser scans.
2.62 % 0.0119 [deg/m] 0.1 s 1 core @ 3.5 Ghz (C/C++)
41 VO3pt
This method uses stereo information.
2.69 % 0.0068 [deg/m] 0.56 s 1 core @ 2.0 Ghz (C/C++)
P. Alcantarilla: Vision Based Localization: From Humanoid Robots to Visually Impaired People. 2011.
P. Alcantarilla, J. Yebes, J. Almazán and L. Bergasa: On Combining Visual SLAM and Dense Scene Flow to Increase the Robustness of Localization and Mapping in Dynamic Environments. ICRA 2012.
42 TGVO
This method uses stereo information.
2.94 % 0.0077 [deg/m] 0.06 s 1 core @ 2.5 Ghz (C/C++)
B. Kitt, A. Geiger and H. Lategahn: Visual Odometry based on Stereo Image Sequences with RANSAC-based Outlier Rejection Scheme. IV 2010.
43 monovo 2.95 % 0.0040 [deg/m] 0.1 s 2 core @ 2.7 Ghz (C/C++)
44 SGPVO
This method uses stereo information.
3.11 % 0.0097 [deg/m] 0.2 s 1 core @ 3.0 Ghz (Matlab)
45 VO3ptLBA
This method uses stereo information.
3.13 % 0.0104 [deg/m] 0.57 s 1 core @ 2.0 Ghz (C/C++)
P. Alcantarilla: Vision Based Localization: From Humanoid Robots to Visually Impaired People. 2011.
P. Alcantarilla, J. Yebes, J. Almazán and L. Bergasa: On Combining Visual SLAM and Dense Scene Flow to Increase the Robustness of Localization and Mapping in Dynamic Environments. ICRA 2012.
46 PLSVO
This method uses stereo information.
3.26 % 0.0095 [deg/m] 0.20 s 2 cores @ 2.5 Ghz (C/C++)
R. Gomez-Ojeda and J. Gonzalez- Jimenez: Robust Stereo Visual Odometry through a Probabilistic Combination of Points and Line Segments. Robotics and Automation (ICRA), 2016 IEEE International Conference on 2016.
47 AvgSLAM
This method uses stereo information.
3.26 % 0.0101 [deg/m] 0.1 s 4 cores @ 2.5 Ghz (C/C++)
48 FMVO 3.61 % 0.0058 [deg/m] 1 s 1 core @ 2.5 Ghz (Python + C/C++)
49 CFORB
This method uses stereo information.
3.73 % 0.0107 [deg/m] 0.9 s 8 cores @ 3.0 Ghz (C/C++)
D. Mankowitz and E. Rivlin: CFORB: Circular FREAK-ORB Visual Odometry. arXiv preprint arXiv:1506.05257 2015.
50 VOFS
This method uses stereo information.
3.94 % 0.0099 [deg/m] 0.51 s 1 core @ 2.0 Ghz (C/C++)
M. Kaess, K. Ni and F. Dellaert: Flow separation for fast and robust stereo odometry. ICRA 2009.
P. Alcantarilla, L. Bergasa and F. Dellaert: Visual Odometry priors for robust EKF-SLAM. ICRA 2010.
51 VOFSLBA
This method uses stereo information.
4.17 % 0.0112 [deg/m] 0.52 s 1 core @ 2.0 Ghz (C/C++)
M. Kaess, K. Ni and F. Dellaert: Flow separation for fast and robust stereo odometry. ICRA 2009.
P. Alcantarilla, L. Bergasa and F. Dellaert: Visual Odometry priors for robust EKF-SLAM. ICRA 2010.
52 BVO 5.36 % 0.0051 [deg/m] 0.1 s 1 core @ 2.0 Ghz (Python + C/C++)
53 EB3DTE+RJMCM 5.50 % 0.0171 [deg/m] 0.64 s 1 core @ 2.5 Ghz (Matlab)
54 EB3DTE 5.52 % 0.0155 [deg/m] 0,64 s 1 cores @ 3.0 Ghz (Matlab)
55 LPF-II 6.13 % 0.0096 [deg/m] 5 s 1 core @ 2.5 Ghz (C/C++)
56 IO
This method uses stereo information.
code 6.55 % 0.0315 [deg/m] 5 s 1 core @ 2.5 Ghz (Matlab)
57 VISO2-M + GP 7.46 % 0.0245 [deg/m] 0.15 s 1 core @ 2.5 Ghz (C/C++)
A. Geiger, J. Ziegler and C. Stiller: StereoScan: Dense 3d Reconstruction in Real-time. IV 2011.
S. Song and M. Chandraker: Robust Scale Estimation in Real-Time Monocular SFM for Autonomous Driving. CVPR 2014.
58 VISO2-M code 11.94 % 0.0234 [deg/m] 0.1 s 1 core @ 2.5 Ghz (C/C++)
A. Geiger, J. Ziegler and C. Stiller: StereoScan: Dense 3d Reconstruction in Real-time. IV 2011.
59 1SCTE 15.08 % 0.0171 [deg/m] 0.02 s 1 core @ 2.5 Ghz (Matlab)
60 OABA 20.95 % 0.0135 [deg/m] 0.5 s 1 core @ 3.5 Ghz (C/C++)
D. Frost, O. Kähler and D. Murray: Object-Aware Bundle Adjustment for Correcting Monocular Scale Drift. Proceedings of the International Conference on Robotics and Automation (ICRA) 2012.
61 NM 88.15 % 0.2090 [deg/m] 1 s 1 core @ 2.5 Ghz (Matlab)
62 ST
This method uses stereo information.
97.70 % 0.2710 [deg/m] 0.2 s 8 cores @ 3.0 Ghz (C/C++)
Table as LaTeX | Only published Methods

Related Datasets

Citation

When using this dataset in your research, we will be happy if you cite us:
@INPROCEEDINGS{Geiger2012CVPR,
  author = {Andreas Geiger and Philip Lenz and Raquel Urtasun},
  title = {Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite},
  booktitle = {Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2012}
}



eXTReMe Tracker