Dataset


The object tracking benchmark consists of 21 training sequences and 29 test sequences. Despite the fact that we have labeled 8 different classes, only the classes 'Car' and 'Pedestrian' are evaluated in our benchmark, as only for those classes enough instances for a comprehensive evaluation have been labeled. The labeling process has been performed in two steps: First we hired a set of annotators, to label 3D bounding boxes as tracklets in point clouds. Since for a pedestrian tracklet, a single 3D bounding box tracklet (dimensions have been fixed) often fits badly, we additionally labeled the left/right boundaries of each object by making use of Mechanical Turk. We also collected labels of the object's occlusion state, and computed the object's truncation via backprojecting a car/pedestrian model into the image plane. We evaluate submitted results using the common metrics CLEAR MOT and MT/PT/ML. Since there is no single ranking criterion, we do not rank methods. Out development kit provides details about the data format as well as utility functions for reading and writing the label files.

Evaluation

The goal in the object tracking task is to estimate object tracklets for the classes 'Car' and 'Pedestrian'. We evaluate 2D 0-based bounding boxes in each image. We like to encourage people to add a confidence measure for every particular frame for this track. For evaluation we only consider detections/objects larger than 25 pixel (height) in the image and do not count Vans as false positives for cars or Sitting Persons as wrong positives for Pedestrians due to their similarity in appearance. As evaluation criterion we follow the CLEARMOT [1] and Mostly-Tracked/Partly-Tracked/Mostly-Lost [2] metrics. We do not rank methods by a single criterion, but bold numbers indicate the best method for a particular metric. To make the methods comparable, the time for object detection is not included in the specified runtime.
[1] K. Bernardin, R. Stiefelhagen: Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics. JIVP 2008.
[2] Y. Li, C. Huang, R. Nevatia: Learning to associate: HybridBoosted multi-target tracker for crowded scene. CVPR 2009.

Additional information used by the methods
  • Stereo: Method uses left and right (stereo) images
  • Laser Points: Method uses point clouds from Velodyne laser scanner
  • GPS: Method uses GPS information
  • Online: Online method (frame-by-frame processing, no latency)

CAR


Method Setting Code MOTA MOTP MT ML IDS FRAG Runtime Environment
DP_MCF code 43.77 % 78.49 % 11.08 % 39.45 % 2738 3241 0.01 s 1 core @ 2.5 Ghz (Matlab)
H. Pirsiavash, D. Ramanan and C. Fowlkes: Globally-Optimal Greedy Algorithms for Tracking a Variable Number of Objects. IEEE conference on Computer Vision and Pattern Recognition (CVPR) 2011.
HM
This is an online method (no batch processing).
41.56 % 78.42 % 7.74 % 42.19 % 12 578 0.01 s 1 core @ 2.5 Ghz (Python)
MCF 43.63 % 78.32 % 10.93 % 40.06 % 23 591 0.01 s 1 core @ 2.5 Ghz (Python + C/C++)
L. Zhang, Y. Li and R. Nevatia: Global data association for multi-object tracking using network flows.. CVPR .
TBD code 51.73 % 78.47 % 13.81 % 34.60 % 33 540 10 s 1 core @ 2.5 Ghz (Matlab + C/C++)
A. Geiger, M. Lauer, C. Wojek, C. Stiller and R. Urtasun: 3D Traffic Scene Understanding from Movable Platforms. Pattern Analysis and Machine Intelligence (PAMI) 2014.
H. Zhang, A. Geiger and R. Urtasun: Understanding High-Level Semantics by Modeling Traffic Patterns. International Conference on Computer Vision (ICCV) 2013.
SSP 53.85 % 77.78 % 21.24 % 27.31 % 7 717 0.6s 1 core @ 2.7 Ghz (Python)
Anonymous submission
mbodSSP
This is an online method (no batch processing).
51.64 % 77.67 % 15.02 % 29.89 % 0 708 0.01 s 1 core @ 2.7 Ghz (Python)
Anonymous submission
DCO code 35.23 % 74.50 % 10.62 % 33.84 % 223 624 0.03 s 1 core @ >3.5 Ghz (Matlab + C/C++)
A. Andriyenko, K. Schindler and S. Roth: Discrete-Continuous Optimization for Multi-Target Tracking. CVPR 2012.
CEM code 47.81 % 77.26 % 14.42 % 33.99 % 125 401 0.09 s 1 core @ >3.5 Ghz (Matlab + C/C++)
A. Milan, S. Roth and K. Schindler: Continuous Energy Minimization for Multitarget Tracking. IEEE TPAMI 2014.
NOMT 62.44 % 78.32 % 31.56 % 27.77 % 13 159 0.09 s 16 core @ 2.5 Ghz (C++)
Anonymous submission
NOMT-HM
This is an online method (no batch processing).
57.55 % 78.79 % 26.86 % 30.50 % 28 253 0.09 s 8 cores @ 2.5 Ghz (Matlab + C/C++)
Anonymous submission
SSP* 66.67 % 78.64 % 40.52 % 8.95 % 194 977 0.6 s 1 core @ 2.7 Ghz (Python)
Anonymous submission
mbodSSP*
This is an online method (no batch processing).
66.66 % 78.83 % 34.29 % 10.47 % 117 894 0.01 s 1 core @ 2.7 Ghz (Python)
Anonymous submission
NOMT-HM*
This is an online method (no batch processing).
69.12 % 80.10 % 38.54 % 15.02 % 109 378 0.09 s 8 cores @ 2.5 Ghz (Matlab + C/C++)
Anonymous submission
MTRM*
This is an online method (no batch processing).
70.52 % 78.45 % 42.34 % 9.56 % 145 576 0.0167 s 1 core @ 4.0 Ghz (Matlab + C/C++)
Anonymous submission
MTRM
This is an online method (no batch processing).
55.11 % 77.81 % 22.31 % 30.96 % 28 401 0.0045 s 1 core @ 4.0 Ghz (Matlab + C/C++)
Anonymous submission
NOMT* 71.68 % 79.55 % 43.10 % 13.96 % 39 236 0.09 s 16 cores @ 2.5 Ghz (C++)
Anonymous submission
RMOT
This is an online method (no batch processing).
49.25 % 75.33 % 15.17 % 33.54 % 51 389 0.01 s 1 core @ 3.5 Ghz (Matlab)
J. Yoon, M. Yang, J. Lim and K. Yoon: Bayesian Multi-Object Tracking Using Motion Context from Multiple Objects. IEEE Winter Conference on Applications of Computer Vision (WACV) 2015.
DCO_X* code 62.82 % 78.96 % 26.10 % 15.33 % 327 996 0.9 s 1 core @ >3.5 Ghz (Matlab + C/C++)
A. Milan, K. Schindler and S. Roth: Detection- and Trajectory-Level Exclusion in Multiple Object Tracking. CVPR 2013.
RMOT*
This is an online method (no batch processing).
60.27 % 75.57 % 27.01 % 11.38 % 216 755 0.02 s 1 core @ 3.5 Ghz (Matlab)
J. Yoon, M. Yang, J. Lim and K. Yoon: Bayesian Multi-Object Tracking Using Motion Context from Multiple Objects. IEEE Winter Conference on Applications of Computer Vision (WACV) 2015.
ODAMOT
This is an online method (no batch processing).
57.34 % 75.45 % 16.69 % 18.97 % 404 1309 1 s 1 core @ 2.5 Ghz (Python)
Anonymous submission
LP_SSVM 57.33 % 77.24 % 27.92 % 23.37 % 18 449 0.06 s 1 core @ 2.5 Ghz (Matlab + C/C++)
Anonymous submission
SSP_SSVM* 70.41 % 77.70 % 41.58 % 9.41 % 73 579 0.01 s 1 core @ 2.5 Ghz (C/C++)
Anonymous submission
FMMOVT 41.91 % 76.83 % 28.38 % 19.88 % 1442 1999 0.05 s 1 core @ 2.5 Ghz (Python)
Anonymous submission
FMMOVT 31.36 % 77.79 % 11.68 % 36.57 % 511 939 0.05 s 1 core @ 2.5 Ghz (C/C++)
Anonymous submission
This table as LaTeX

PEDESTRIAN


Method Setting Code MOTA MOTP MT ML IDS FRAG Runtime Environment
CEM code 36.21 % 74.55 % 7.95 % 53.04 % 221 1011 0.09 s 1 core @ >3.5 Ghz (Matlab + C/C++)
A. Milan, S. Roth and K. Schindler: Continuous Energy Minimization for Multitarget Tracking. IEEE TPAMI 2014.
NOMT 47.84 % 75.01 % 14.54 % 42.71 % 47 959 0.09 s 16 core @ 2.5 Ghz (C++)
Anonymous submission
NOMT-HM
This is an online method (no batch processing).
41.67 % 75.77 % 11.43 % 51.65 % 101 996 0.09 s 8 cores @ 2.5 Ghz (Matlab + C/C++)
Anonymous submission
NOMT-HM*
This is an online method (no batch processing).
54.46 % 77.51 % 17.31 % 42.32 % 295 1248 0.09 s 8 cores @ 2.5 Ghz (Matlab + C/C++)
Anonymous submission
MTRM*
This is an online method (no batch processing).
57.64 % 76.22 % 17.67 % 38.86 % 290 1487 0.0167 s 1 core @ 4.0 Ghz (Matlab + C/C++)
Anonymous submission
MTRM
This is an online method (no batch processing).
43.05 % 74.71 % 8.67 % 46.50 % 85 1288 0.0045 s 1 core @ 4.0 Ghz (Matlab + C/C++)
Anonymous submission
NOMT* 58.80 % 77.10 % 23.52 % 34.76 % 102 908 0.09 s 16 cores @ 2.5 Ghz (C++)
Anonymous submission
RMOT
This is an online method (no batch processing).
39.94 % 72.86 % 10.02 % 47.54 % 132 1081 0.01 s 1 core @ 3.5 Ghz (Matlab)
J. Yoon, M. Yang, J. Lim and K. Yoon: Bayesian Multi-Object Tracking Using Motion Context from Multiple Objects. IEEE Winter Conference on Applications of Computer Vision (WACV) 2015.
RMOT*
This is an online method (no batch processing).
51.06 % 74.19 % 16.93 % 41.28 % 372 1515 0.02 s 1 core @ 3.5 Ghz (Matlab)
J. Yoon, M. Yang, J. Lim and K. Yoon: Bayesian Multi-Object Tracking Using Motion Context from Multiple Objects. IEEE Winter Conference on Applications of Computer Vision (WACV) 2015.
LP_SSVM 43.66 % 74.14 % 10.41 % 43.38 % 87 1291 0.06 s 1 core @ 2.5 Ghz (Matlab + C/C++)
Anonymous submission
SSP_SSVM* 56.54 % 75.44 % 18.70 % 33.71 % 181 1448 0.01 s 1 core @ 2.5 Ghz (C/C++)
Anonymous submission
This table as LaTeX

Related Datasets

  • TUD Datasets: "TUD Multiview Pedestrians" and "TUD Stadmitte" Datasets.
  • PETS 2009: The Datasets for the "Performance Evaluation of Tracking and Surveillance"" Workshop.
  • EPFL Terrace: Multi-camera pedestrian videos.
  • ETHZ Sequences: Inner City Sequences from Mobile Platforms.



eXTReMe Tracker