Dataset


The visual odometry benchmark consists of 22 stereo sequences, saved in loss less png format: We provide 11 sequences (00-10) with ground truth trajectories for training and 11 sequences (11-21) without ground truth for evaluation. For this benchmark you may provide results using monocular/stereo visual odometry or SLAM. The only restriction we impose is that your method is fully automatic (e.g., no manual loop-closure tagging is allowed) and that the same parameter set is used for all sequences. A development kit provides details about the data format.

Evaluation

From all test sequences, our evaluation computes translational and rotational errors for all possible subsequences of length (5,10,50,100,150,...,400) meters. The evaluation table below ranks methods according to the average of those values, where errors are measured in percent (for translation) and in degrees per meter (for rotation). A more detailed comparison for different trajectory lengths and driving speeds can be found in the plots underneath.

  • Stereo: Method uses left and right (stereo) images
  • Flow: Method uses optical flow (2 temporally adjacent images)
  • Multiview: Method uses more than 2 temporally adjacent images
  • Laser Points: Method uses point clouds from Velodyne laser scanner
  • GPS: Method uses GPS information
  • Motion stereo: Method uses epipolar geometry for computing optical flow

Rank Method Setting Code Translation Rotation Runtime Environment
1 MFI 1.62 % 0.0062 [deg/m] 0.1 s 4 cores @ 3.0 Ghz (C/C++)
Anonymous submission
2 VoBa 1.77 % 0.0066 [deg/m] 0.1 s 1 core @ 2.0 Ghz (C/C++)
3 VISO2-S code 1.83 % 0.0152 [deg/m] 0.05 s 1 core @ 2.5 Ghz (C/C++)
A. Geiger, J. Ziegler and C. Stiller: StereoScan: Dense 3d Reconstruction in Real-time. IV 2011.
4 SSLAM 1.87 % 0.0083 [deg/m] 0.5 s 8 cores @ 3.5 Ghz (C/C++)
Anonymous submission
5 eVO 1.93 % 0.0076 [deg/m] 0.05 s 2 cores @ 2.0 Ghz (C/C++)
Anonymous submission
6 D6DVO 2.10 % 0.0083 [deg/m] 0.03 s 1 core @ 2.5 Ghz (C/C++)
A. Comport, E. Malis and P. Rives: Accurate Quadrifocal Tracking for Robust 3D Visual Odometry. ICRA 2007.
M. Meilland, A. Comport and P. Rives: Dense visual mapping of large scale environments for real-time localisation. ICRA 2011.
7 GT_VO3pt 2.21 % 0.0117 [deg/m] 1.26 s 1 core @ 2.5 Ghz (C/C++)
C. Beall, B. Lawrence, V. Ila and F. Dellaert: 3D reconstruction of underwater structures. IROS 2010.
8 BoofCV-SQ3 code 2.27 % 0.0111 [deg/m] 0.14 s 1 core @ 2.5 Ghz (Java)
9 MICP_VO 2.35 % 0.0102 [deg/m] 0.01 s 1 core @ 2.5 Ghz (C++)
Anonymous submission
10 TGVO 2.44 % 0.0105 [deg/m] 0.06 s 1 core @ 2.5 Ghz (C/C++)
B. Kitt, A. Geiger and H. Lategahn: Visual Odometry based on Stereo Image Sequences with RANSAC-based Outlier Rejection Scheme. IV 2010.
11 SVO 2.45 % 0.0109 [deg/m] 0.05 s 2 cores @ 2.5 Ghz (C/C++)
Anonymous submission
12 SSLAM-HR 2.45 % 0.0112 [deg/m] 0.5 s 8 cores @ 3.5 Ghz (C/C++)
Anonymous submission
13 VO3pt 2.93 % 0.0116 [deg/m] 0.56 s 1 core @ 2.0 Ghz (C/C++)
P. Alcantarilla: Vision Based Localization: From Humanoid Robots to Visually Impaired People. 2011.
P. Alcantarilla, J. Yebes, J. Almazán and L. Bergasa: On Combining Visual SLAM and Dense Scene Flow to Increase the Robustness of Localization and Mapping in Dynamic Environments. ICRA 2012.
14 VO3ptLBA 3.17 % 0.0180 [deg/m] 0.57 s 1 core @ 2.0 Ghz (C/C++)
P. Alcantarilla: Vision Based Localization: From Humanoid Robots to Visually Impaired People. 2011.
P. Alcantarilla, J. Yebes, J. Almazán and L. Bergasa: On Combining Visual SLAM and Dense Scene Flow to Increase the Robustness of Localization and Mapping in Dynamic Environments. ICRA 2012.
15 MSD VO 3.50 % 0.0166 [deg/m] 0.07 s 1 core @ 2.8 Ghz (C/C++)
Anonymous submission
16 MLM-SFM 4.07 % 0.0104 [deg/m] 0.03 s 5 cores @ 2.5 Ghz (C/C++)
Anonymous submission
17 VOFS 4.21 % 0.0158 [deg/m] 0.51 s 1 core @ 2.0 Ghz (C/C++)
M. Kaess, K. Ni and F. Dellaert: Flow separation for fast and robust stereo odometry. ICRA 2009.
P. Alcantarilla, L. Bergasa and F. Dellaert: Visual Odometry priors for robust EKF-SLAM. ICRA 2010.
18 VOFSLBA 4.35 % 0.0189 [deg/m] 0.52 s 1 core @ 2.0 Ghz (C/C++)
M. Kaess, K. Ni and F. Dellaert: Flow separation for fast and robust stereo odometry. ICRA 2009.
P. Alcantarilla, L. Bergasa and F. Dellaert: Visual Odometry priors for robust EKF-SLAM. ICRA 2010.
19 VISO2-M code 13.79 % 0.0372 [deg/m] 0.1 s 1 core @ 2.5 Ghz (C/C++)
A. Geiger, J. Ziegler and C. Stiller: StereoScan: Dense 3d Reconstruction in Real-time. IV 2011.
This table as LaTeX


This figure as: png eps pdf txt gnuplot



This figure as: png eps pdf txt gnuplot



This figure as: png eps pdf txt gnuplot



This figure as: png eps pdf txt gnuplot


Related Datasets

  • NYU RGB-D Dataset: Indoor dataset captured with a Microsoft Kinect that provides semantic labels.
  • TUM RGB-D Dataset: Indoor dataset captured with Microsoft Kinect and high-accuracy motion capturing.
  • New College Dataset: 30 GB of data for 6 D.O.F. navigation and mapping (metric or topological) using vision and/or laser.
  • The Rawseeds Project: Indoor and outdoor datasets with GPS, odometry, stereo, omnicam and laser measurements for visual, laser-based, omnidirectional, sonar and multi-sensor SLAM evaluation.
  • Victoria Park Sequence: Widely used sequence for evaluating laser-based SLAM. Trees serve as landmarks, detection code is included.
  • Malaga Dataset 2009: Dataset with GPS, Cameras and 3D laser information, recorded in the city of Malaga, Spain.
  • Ford Campus Vision and Lidar Dataset: Dataset collected by a Ford F-250 pickup, equipped with IMU, Velodyne and Ladybug.



eXTReMe Tracker