Andreas Geiger (KIT) | Philip Lenz (KIT) | Christoph Stiller (KIT) | Raquel Urtasun (TTI-C)

Dataset


The visual odometry benchmark consists of 22 stereo sequences, saved in loss less png format: We provide 11 sequences (00-10) with ground truth trajectories for training and 11 sequences (11-21) without ground truth for evaluation. For this benchmark you may provide results using monocular/stereo visual odometry or SLAM. The only restriction we impose is that your method is fully automatic (e.g., no manual loop-closure tagging is allowed) and that the same parameter set is used for all sequences. A development kit provides details about the data format.

Evaluation

From all test sequences, our evaluation computes translational and rotational errors for all possible subsequences of length (5,10,50,100,150,...,400) meters. The evaluation table below ranks methods according to the average of those values, where errors are measured in percent (for translation) and in degrees per meter (for rotation). A more detailed comparison for different trajectory lengths and driving speeds can be found in the plots underneath.

Rank Method Setting Code Translation Rotation Runtime Environment
1 MFI 1.73 % 0.0067 [deg/m] 0.1 s 4 cores @ 3.0 Ghz (C/C++)
Anonymous submission
2 VoBa 1.77 % 0.0066 [deg/m] 0.1 s 1 core @ 2.0 Ghz (C/C++)
3 VISO2-SAM 1.83 % 0.0152 [deg/m] 0.05 s 1 core @ 3.5 Ghz (C/C++)
Anonymous submission
4 eVO 1.93 % 0.0076 [deg/m] 0.05 s 2 cores @ 2.0 Ghz (C/C++)
Anonymous submission
5 D6DVO 2.10 % 0.0083 [deg/m] 0.03 s 1 core @ 2.5 Ghz (C/C++)
A. Comport, E. Malis and P. Rives: Accurate Quadrifocal Tracking for Robust 3D Visual Odometry. ICRA 2007.
M. Meilland, A. Comport and P. Rives: Dense visual mapping of large scale environments for real-time localisation. ICRA 2011.
6 GT_VO3pt 2.21 % 0.0117 [deg/m] 1.26 s 1 core @ 2.5 Ghz (C/C++)
C. Beall, B. Lawrence, V. Ila and F. Dellaert: 3D reconstruction of underwater structures. IROS 2010.
7 VISO2-S code 2.27 % 0.0152 [deg/m] 0.05 s 1 core @ 2.5 Ghz (C/C++)
A. Geiger, J. Ziegler and C. Stiller: StereoScan: Dense 3d Reconstruction in Real-time. IV 2011.
8 BoofCV-SQ3 code 2.27 % 0.0111 [deg/m] 0.14 s 1 core @ 2.5 Ghz (Java)
9 TGVO 2.44 % 0.0105 [deg/m] 0.06 s 1 core @ 2.5 Ghz (C/C++)
B. Kitt, A. Geiger and H. Lategahn: Visual Odometry based on Stereo Image Sequences with RANSAC-based Outlier Rejection Scheme. IV 2010.
10 SVO 2.45 % 0.0109 [deg/m] 0.05 s 2 cores @ 2.5 Ghz (C/C++)
Anonymous submission
11 SSLAM 2.45 % 0.0112 [deg/m] 0.5 s 8 cores @ 3.5 Ghz (C/C++)
Anonymous submission
12 VO3pt 2.93 % 0.0116 [deg/m] 0.56 s 1 core @ 2.0 Ghz (C/C++)
P. Alcantarilla: Vision Based Localization: From Humanoid Robots to Visually Impaired People. 2011.
P. Alcantarilla, J. Yebes, J. Almazán and L. Bergasa: On Combining Visual SLAM and Dense Scene Flow to Increase the Robustness of Localization and Mapping in Dynamic Environments. ICRA 2012.
13 VO3ptLBA 3.17 % 0.0180 [deg/m] 0.57 s 1 core @ 2.0 Ghz (C/C++)
P. Alcantarilla: Vision Based Localization: From Humanoid Robots to Visually Impaired People. 2011.
P. Alcantarilla, J. Yebes, J. Almazán and L. Bergasa: On Combining Visual SLAM and Dense Scene Flow to Increase the Robustness of Localization and Mapping in Dynamic Environments. ICRA 2012.
14 MSD VO 3.50 % 0.0166 [deg/m] 0.07 s 1 core @ 2.8 Ghz (C/C++)
Anonymous submission
15 KPnP 3.99 % 0.0138 [deg/m] 0.2 s 1 core @ 2.5 Ghz (Matlab)
Anonymous submission
16 MLM-SFM 4.07 % 0.0104 [deg/m] 0.03 s 5 cores @ 2.5 Ghz (C/C++)
Anonymous submission
17 VOFS 4.21 % 0.0158 [deg/m] 0.51 s 1 core @ 2.0 Ghz (C/C++)
M. Kaess, K. Ni and F. Dellaert: Flow separation for fast and robust stereo odometry. ICRA 2009.
P. Alcantarilla, L. Bergasa and F. Dellaert: Visual Odometry priors for robust EKF-SLAM. ICRA 2010.
18 VOFSLBA 4.35 % 0.0189 [deg/m] 0.52 s 1 core @ 2.0 Ghz (C/C++)
M. Kaess, K. Ni and F. Dellaert: Flow separation for fast and robust stereo odometry. ICRA 2009.
P. Alcantarilla, L. Bergasa and F. Dellaert: Visual Odometry priors for robust EKF-SLAM. ICRA 2010.
19 VISO2-M code 13.79 % 0.0372 [deg/m] 0.1 s 1 core @ 2.5 Ghz (C/C++)
A. Geiger, J. Ziegler and C. Stiller: StereoScan: Dense 3d Reconstruction in Real-time. IV 2011.
This table as LaTeX


This figure as: png eps pdf txt gnuplot



This figure as: png eps pdf txt gnuplot



This figure as: png eps pdf txt gnuplot



This figure as: png eps pdf txt gnuplot


Related Datasets

  • NYU RGB-D Dataset: Indoor dataset captured with a Microsoft Kinect that provides semantic labels.
  • TUM RGB-D Dataset: Indoor dataset captured with Microsoft Kinect and high-accuracy motion capturing.
  • New College Dataset: 30 GB of data for 6 D.O.F. navigation and mapping (metric or topological) using vision and/or laser.
  • The Rawseeds Project: Indoor and outdoor datasets with GPS, odometry, stereo, omnicam and laser measurements for visual, laser-based, omnidirectional, sonar and multi-sensor SLAM evaluation.
  • Victoria Park Sequence: Widely used sequence for evaluating laser-based SLAM. Trees serve as landmarks, detection code is included.
  • Malaga Dataset 2009: Dataset with GPS, Cameras and 3D laser information, recorded in the city of Malaga, Spain.
  • Ford Campus Vision and Lidar Dataset: Dataset collected by a Ford F-250 pickup, equipped with IMU, Velodyne and Ladybug.



eXTReMe Tracker