3D Object Detection Evaluation 2017


The 3D object detection benchmark consists of 7481 training images and 7518 test images as well as the corresponding point clouds, comprising a total of 80.256 labeled objects. For evaluation, we compute precision-recall curves. To rank the methods we compute average precision. We require that all methods use the same parameter set for all test pairs. Our development kit provides details about the data format as well as MATLAB / C++ utility functions for reading and writing the label files.

We evaluate 3D object detection performance using the PASCAL criteria also used for 2D object detection. Far objects are thus filtered based on their bounding box height in the image plane. As only objects also appearing on the image plane are labeled, objects in don't car areas do not count as false positives. We note that the evaluation does not take care of ignoring detections that are not visible on the image plane — these detections might give rise to false positives. For cars we require an 3D bounding box overlap of 70%, while for pedestrians and cyclists we require a 3D bounding box overlap of 50%. Difficulties are defined as follows:

  • Easy: Min. bounding box height: 40 Px, Max. occlusion level: Fully visible, Max. truncation: 15 %
  • Moderate: Min. bounding box height: 25 Px, Max. occlusion level: Partly occluded, Max. truncation: 30 %
  • Hard: Min. bounding box height: 25 Px, Max. occlusion level: Difficult to see, Max. truncation: 50 %

All methods are ranked based on the moderately difficult results.

Additional information used by the methods
  • Stereo: Method uses left and right (stereo) images
  • Flow: Method uses optical flow (2 temporally adjacent images)
  • Multiview: Method uses more than 2 temporally adjacent images
  • Laser Points: Method uses point clouds from Velodyne laser scanner
  • Additional training data: Use of additional data sources for training (see details)

Car


Method Setting Code Moderate Easy Hard Runtime Environment
1 VxNet++(LiDAR)
This method makes use of Velodyne laser scans.
73.66 % 83.13 % 66.20 % 0.06 s GPU @ 1.8 Ghz (Python)
2 AVOD-FPN
This method makes use of Velodyne laser scans.
code 71.88 % 81.94 % 66.38 % 0.1 s Titan X (Pascal)
J. Ku, M. Mozifian, J. Lee, A. Harakeh and S. Waslander: Joint 3D Proposal Generation and Object Detection from View Aggregation. arXiv preprint arXiv:1712.02294 2017.
3 F-PointNet
This method makes use of Velodyne laser scans.
code 70.39 % 81.20 % 62.19 % 0.17 s GPU @ 3.0 Ghz (Python)
C. Qi, W. Liu, C. Wu, H. Su and L. Guibas: Frustum PointNets for 3D Object Detection from RGB-D Data. arXiv preprint arXiv:1711.08488 2017.
4 DF-PC_CNN
This method makes use of Velodyne laser scans.
66.22 % 80.28 % 58.94 % 0.5 s GPU @ 3.0 Ghz (Matlab + C/C++)
5 AVOD
This method makes use of Velodyne laser scans.
code 65.78 % 73.59 % 58.38 % 0.08 s Titan X (pascal)
J. Ku, M. Mozifian, J. Lee, A. Harakeh and S. Waslander: Joint 3D Proposal Generation and Object Detection from View Aggregation. arXiv preprint arXiv:1712.02294 2017.
6 VxNet(LiDAR)
This method makes use of Velodyne laser scans.
65.11 % 77.47 % 57.73 % 0.03 s GPU @ 2.5 Ghz (Python + C/C++)
7 LTT 64.83 % 72.27 % 57.26 % 0.4 s 1 core @ 3.5 Ghz (Python)
8 Kiwoo 64.21 % 75.62 % 57.19 % 0.1 s 1 core @ 2.5 Ghz (Python + C/C++)
9 AVOD-SSD
This method makes use of Velodyne laser scans.
code 63.87 % 73.64 % 56.90 % 0.09 s GPU @ 2.5 Ghz (Python)
10 MV3D
This method makes use of Velodyne laser scans.
62.35 % 71.09 % 55.12 % 0.36 s GPU @ 2.5 Ghz (Python + C/C++)
X. Chen, H. Ma, J. Wan, B. Li and T. Xia: Multi-View 3D Object Detection Network for Autonomous Driving. CVPR 2017.
11 T2Method 62.08 % 74.36 % 55.14 % 0.05 s GPU @ 2.5 Ghz (Python + C/C++)
12 A3DODWTDA
This method makes use of Velodyne laser scans.
56.81 % 59.35 % 50.51 % 0.08 s GPU @ 3.0 Ghz (Python)
13 tester 53.20 % 65.56 % 48.59 % 0.1
14 MV3D (LIDAR)
This method makes use of Velodyne laser scans.
52.73 % 66.77 % 51.31 % 0.24 s GPU @ 2.5 Ghz (Python + C/C++)
X. Chen, H. Ma, J. Wan, B. Li and T. Xia: Multi-View 3D Object Detection Network for Autonomous Driving. CVPR 2017.
15 F-PC_CNN
This method makes use of Velodyne laser scans.
48.07 % 60.06 % 45.22 % 0.5 s GPU @ 3.0 Ghz (Matlab + C/C++)
X. Du, M. Jr., S. Karaman and D. Rus: A General Pipeline for 3D Detection of Vehicles. 2018.
16 Roadstar.ai 44.00 % 48.60 % 40.05 % 0.08 s GPU @ 2.0 Ghz (Python)
17 SDN
This method makes use of Velodyne laser scans.
24.08 % 37.87 % 22.01 % 0.096 s GPU @ 1.7 Ghz (Python)
18 RT3D
This method makes use of Velodyne laser scans.
21.27 % 23.49 % 19.81 % 0.09 s GPU @ 1.8Ghz
19 LMnet
This method makes use of Velodyne laser scans.
15.67 % 19.20 % 15.83 % 0.013 s GPU @ 1.1 Ghz (Python + C/C++)
20 LMNetV2
This method makes use of Velodyne laser scans.
15.24 % 14.75 % 12.85 % 0.02 s GPU @ 2.5 Ghz (C/C++)
K. Minemura, H. Liau, A. Monrroy and S. Kato: LMNet: Real-time Multiclass Object Detection on CPU using 3D LiDAR. 2018.
21 3dSSD 14.97 % 14.71 % 19.43 % 0.03 s GPU @ 2.5 Ghz (Python + C/C++)
22 BirdNet
This method makes use of Velodyne laser scans.
13.44 % 14.75 % 12.04 % 0.11 s Titan Xp GPU
J. Beltran, C. Guindel, F. Moreno, D. Cruzado, F. Garcia and A. Escalera: BirdNet: a 3D Object Detection Framework from LiDAR information. arXiv preprint arXiv:1805.01195 2018.
23 Licar
This method makes use of Velodyne laser scans.
12.88 % 16.25 % 13.67 % 0.09 s GPU @ 2.0 Ghz (Python)
24 LMnetV1.1 11.51 % 20.52 % 11.53 % 0.01 s GPU @ 1.0 Ghz (Python + C/C++)
25 TopNet-HighRes
This method makes use of Velodyne laser scans.
11.50 % 13.15 % 11.38 % 0.27 s NVIDIA GeForce 1080 Ti (tensorflow-gpu)
S. Wirges, T. Fischer, J. Frias and C. Stiller: Object Detection and Classification in Occupancy Grid Maps using Deep Convolutional Networks. 2018.
26 SAITv1 11.01 % 12.92 % 10.45 % 0.18 s GPU @ 2.5 Ghz (Python, C/C++)
27 TopNet-DecayRate
This method makes use of Velodyne laser scans.
10.72 % 11.27 % 10.47 % 0.12 s NVIDIA GeForce 1080 Ti (tensorflow-gpu)
S. Wirges, T. Fischer, J. Frias and C. Stiller: Object Detection and Classification in Occupancy Grid Maps using Deep Convolutional Networks. 2018.
28 M3D 7.81 % 10.25 % 6.54 % 0.4 s GPU @ 2.5 Ghz (Python + C/C++)
29 DoBEM 6.95 % 7.42 % 13.45 % 0.6 s GPU @ 2.5 Ghz (Python + C/C++)
S. Yu, T. Westfechtel, R. Hamada, K. Ohno and S. Tadokoro: Vehicle Detection and Localization on Bird's Eye View Elevation Images Using Convolutional Neural Network. IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR) 2017.
30 CSoR
This method makes use of Velodyne laser scans.
6.79 % 6.76 % 6.14 % 3.5 s 4 cores @ >3.5 Ghz (Python + C/C++)
L. Plotkin: PyDriver: Entwicklung eines Frameworks für räumliche Detektion und Klassifikation von Objekten in Fahrzeugumgebung. 2015.
31 A3DODWTDA (image) 6.45 % 6.76 % 4.87 % 0.8 s GPU @ 3.0 Ghz (Python)
32 MonoFusion 5.18 % 7.08 % 4.68 % 0.12 s TITAN X GPU
33 3D-SSMFCNN code 2.28 % 2.39 % 1.52 % 0.1 s GPU @ 1.5 Ghz (C/C++)
L. Novak: Vehicle Detection and Pose Estimation for Autonomous Driving. 2017.
34 3DVSSD 1.14 % 1.38 % 1.27 % 0.06 s 1 core @ 2.5 Ghz (C/C++)
35 SPC
This method makes use of Velodyne laser scans.
0.52 % 0.68 % 0.60 % 0.4 s 4 cores @ 2.5 Ghz (Python)
36 LidarNet
This method makes use of Velodyne laser scans.
0.02 % 0.01 % 0.03 % 0.007 s GPU @ 2.5 Ghz (C/C++)
37 mBoW
This method makes use of Velodyne laser scans.
0.00 % 0.00 % 0.00 % 10 s 1 core @ 2.5 Ghz (C/C++)
J. Behley, V. Steinhage and A. Cremers: Laser-based Segment Classification Using a Mixture of Bag-of-Words. Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2013.
Table as LaTeX | Only published Methods

Pedestrian


Method Setting Code Moderate Easy Hard Runtime Environment
1 F-PointNet
This method makes use of Velodyne laser scans.
code 44.89 % 51.21 % 40.23 % 0.17 s GPU @ 3.0 Ghz (Python)
C. Qi, W. Liu, C. Wu, H. Su and L. Guibas: Frustum PointNets for 3D Object Detection from RGB-D Data. arXiv preprint arXiv:1711.08488 2017.
2 AVOD-FPN
This method makes use of Velodyne laser scans.
code 42.81 % 50.80 % 40.88 % 0.1 s Titan X (Pascal)
J. Ku, M. Mozifian, J. Lee, A. Harakeh and S. Waslander: Joint 3D Proposal Generation and Object Detection from View Aggregation. arXiv preprint arXiv:1712.02294 2017.
3 VxNet++(LiDAR)
This method makes use of Velodyne laser scans.
42.56 % 51.07 % 37.29 % 0.06 s GPU @ 1.8 Ghz (Python)
4 VxNet(LiDAR)
This method makes use of Velodyne laser scans.
33.69 % 39.48 % 31.51 % 0.03 s GPU @ 2.5 Ghz (Python + C/C++)
5 AVOD
This method makes use of Velodyne laser scans.
code 31.51 % 38.28 % 26.98 % 0.08 s Titan X (pascal)
J. Ku, M. Mozifian, J. Lee, A. Harakeh and S. Waslander: Joint 3D Proposal Generation and Object Detection from View Aggregation. arXiv preprint arXiv:1712.02294 2017.
6 Roadstar.ai 23.28 % 24.61 % 21.97 % 0.08 s GPU @ 2.0 Ghz (Python)
7 3dSSD 17.35 % 20.22 % 17.20 % 0.03 s GPU @ 2.5 Ghz (Python + C/C++)
8 BirdNet
This method makes use of Velodyne laser scans.
11.80 % 14.31 % 10.55 % 0.11 s Titan Xp GPU
J. Beltran, C. Guindel, F. Moreno, D. Cruzado, F. Garcia and A. Escalera: BirdNet: a 3D Object Detection Framework from LiDAR information. arXiv preprint arXiv:1805.01195 2018.
9 LMNetV2
This method makes use of Velodyne laser scans.
11.46 % 13.64 % 11.57 % 0.02 s GPU @ 2.5 Ghz (C/C++)
K. Minemura, H. Liau, A. Monrroy and S. Kato: LMNet: Real-time Multiclass Object Detection on CPU using 3D LiDAR. 2018.
10 TopNet-HighRes
This method makes use of Velodyne laser scans.
10.68 % 11.01 % 10.63 % 0.27 s NVIDIA GeForce 1080 Ti (tensorflow-gpu)
S. Wirges, T. Fischer, J. Frias and C. Stiller: Object Detection and Classification in Occupancy Grid Maps using Deep Convolutional Networks. 2018.
11 TopNet-DecayRate
This method makes use of Velodyne laser scans.
1.01 % 1.05 % 1.05 % 0.12 s NVIDIA GeForce 1080 Ti (tensorflow-gpu)
S. Wirges, T. Fischer, J. Frias and C. Stiller: Object Detection and Classification in Occupancy Grid Maps using Deep Convolutional Networks. 2018.
12 LMnet
This method makes use of Velodyne laser scans.
0.70 % 0.66 % 0.70 % 0.013 s GPU @ 1.1 Ghz (Python + C/C++)
13 LMnetV1.1 0.69 % 0.64 % 0.74 % 0.01 s GPU @ 1.0 Ghz (Python + C/C++)
14 mBoW
This method makes use of Velodyne laser scans.
0.00 % 0.00 % 0.00 % 10 s 1 core @ 2.5 Ghz (C/C++)
J. Behley, V. Steinhage and A. Cremers: Laser-based Segment Classification Using a Mixture of Bag-of-Words. Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2013.
Table as LaTeX | Only published Methods

Cyclist


Method Setting Code Moderate Easy Hard Runtime Environment
1 F-PointNet
This method makes use of Velodyne laser scans.
code 56.77 % 71.96 % 50.39 % 0.17 s GPU @ 3.0 Ghz (Python)
C. Qi, W. Liu, C. Wu, H. Su and L. Guibas: Frustum PointNets for 3D Object Detection from RGB-D Data. arXiv preprint arXiv:1711.08488 2017.
2 VxNet++(LiDAR)
This method makes use of Velodyne laser scans.
53.85 % 70.51 % 46.90 % 0.06 s GPU @ 1.8 Ghz (Python)
3 AVOD-FPN
This method makes use of Velodyne laser scans.
code 52.18 % 64.00 % 46.61 % 0.1 s Titan X (Pascal)
J. Ku, M. Mozifian, J. Lee, A. Harakeh and S. Waslander: Joint 3D Proposal Generation and Object Detection from View Aggregation. arXiv preprint arXiv:1712.02294 2017.
4 VxNet(LiDAR)
This method makes use of Velodyne laser scans.
48.36 % 61.22 % 44.37 % 0.03 s GPU @ 2.5 Ghz (Python + C/C++)
5 Roadstar.ai 45.51 % 55.85 % 41.48 % 0.08 s GPU @ 2.0 Ghz (Python)
6 AVOD
This method makes use of Velodyne laser scans.
code 44.90 % 60.11 % 38.80 % 0.08 s Titan X (pascal)
J. Ku, M. Mozifian, J. Lee, A. Harakeh and S. Waslander: Joint 3D Proposal Generation and Object Detection from View Aggregation. arXiv preprint arXiv:1712.02294 2017.
7 TopNet-HighRes
This method makes use of Velodyne laser scans.
15.42 % 20.35 % 15.60 % 0.27 s NVIDIA GeForce 1080 Ti (tensorflow-gpu)
S. Wirges, T. Fischer, J. Frias and C. Stiller: Object Detection and Classification in Occupancy Grid Maps using Deep Convolutional Networks. 2018.
8 TopNet-DecayRate
This method makes use of Velodyne laser scans.
12.51 % 17.62 % 12.75 % 0.12 s NVIDIA GeForce 1080 Ti (tensorflow-gpu)
S. Wirges, T. Fischer, J. Frias and C. Stiller: Object Detection and Classification in Occupancy Grid Maps using Deep Convolutional Networks. 2018.
9 BirdNet
This method makes use of Velodyne laser scans.
12.43 % 18.35 % 11.88 % 0.11 s Titan Xp GPU
J. Beltran, C. Guindel, F. Moreno, D. Cruzado, F. Garcia and A. Escalera: BirdNet: a 3D Object Detection Framework from LiDAR information. arXiv preprint arXiv:1805.01195 2018.
10 LMNetV2
This method makes use of Velodyne laser scans.
3.23 % 2.84 % 3.28 % 0.02 s GPU @ 2.5 Ghz (C/C++)
K. Minemura, H. Liau, A. Monrroy and S. Kato: LMNet: Real-time Multiclass Object Detection on CPU using 3D LiDAR. 2018.
11 LMnetV1.1 0.31 % 0.49 % 0.55 % 0.01 s GPU @ 1.0 Ghz (Python + C/C++)
12 LMnet
This method makes use of Velodyne laser scans.
0.29 % 0.55 % 0.36 % 0.013 s GPU @ 1.1 Ghz (Python + C/C++)
13 3dSSD 0.24 % 0.25 % 0.25 % 0.03 s GPU @ 2.5 Ghz (Python + C/C++)
14 mBoW
This method makes use of Velodyne laser scans.
0.00 % 0.00 % 0.00 % 10 s 1 core @ 2.5 Ghz (C/C++)
J. Behley, V. Steinhage and A. Cremers: Laser-based Segment Classification Using a Mixture of Bag-of-Words. Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2013.
Table as LaTeX | Only published Methods

Related Datasets

Citation

When using this dataset in your research, we will be happy if you cite us:
@INPROCEEDINGS{Geiger2012CVPR,
  author = {Andreas Geiger and Philip Lenz and Raquel Urtasun},
  title = {Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite},
  booktitle = {Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2012}
}



eXTReMe Tracker