The KITTI Vision Benchmark Suite

Bird's Eye View Evaluation 2017

The bird's eye view benchmark consists of 7481 training images and 7518 test images as well as the corresponding point clouds, comprising a total of 80.256 labeled objects. For evaluation, we compute precision-recall curves. To rank the methods we compute average precision. We require that all methods use the same parameter set for all test pairs. Our development kit provides details about the data format as well as MATLAB / C++ utility functions for reading and writing the label files.

Download left color images of object data set (12 GB)
Download right color images, if you want to use stereo information (12 GB)
Download the 3 temporally preceding frames (left color) (36 GB)
Download the 3 temporally preceding frames (right color) (36 GB)
Download Velodyne point clouds, if you want to use laser information (29 GB)
Download camera calibration matrices of object data set (16 MB)
Download training labels of object data set (5 MB)
Download object development kit (1 MB) (including 3D object detection and bird's eye view evaluation code)
Download pre-trained LSVM baseline models (5 MB) used in Joint 3D Estimation of Objects and Scene Layout (NIPS 2011). These models are referred to as LSVM-MDPM-sv (supervised version) and LSVM-MDPM-us (unsupervised version) in the tables below.
Download reference detections (L-SVM) for training and test set (800 MB)
Qianli Liao (NYU) has put together code to convert from KITTI to PASCAL VOC file format (documentation included, requires Emacs).
Karl Rosaen (U.Mich) has released code to convert between KITTI, KITTI tracking, Pascal VOC, Udacity, CrowdAI and AUTTI formats.
We thank David Stutz and Bo Li for developing the 3D object detection benchmark.

We evaluate bird's eye view detection performance using the PASCAL criteria also used for 2D object detection. Far objects are thus filtered based on their bounding box height in the image plane. As only objects also appearing on the image plane are labeled, objects in don't car areas do not count as false positives. We note that the evaluation does not take care of ignoring detections that are not visible on the image plane — these detections might give rise to false positives. For cars we require a bounding box overlap of 70% in bird's eye view, while for pedestrians and cyclists we require an overlap of 50%. Difficulties are defined as follows:

Easy: Min. bounding box height: 40 Px, Max. occlusion level: Fully visible, Max. truncation: 15 %
Moderate: Min. bounding box height: 25 Px, Max. occlusion level: Partly occluded, Max. truncation: 30 %
Hard: Min. bounding box height: 25 Px, Max. occlusion level: Difficult to see, Max. truncation: 50 %

All methods are ranked based on the moderately difficult results.

Important Policy Update: As more and more non-published work and re-implementations of existing work is submitted to KITTI, we have established a new policy: from now on, only submissions with significant novelty that are leading to a peer-reviewed paper in a conference or journal are allowed. Minor modifications of existing algorithms or student research projects are not allowed. Such work must be evaluated on a split of the training set. To ensure that our policy is adopted, new users must detail their status, describe their work and specify the targeted venue during registration. Furthermore, we will regularly delete all entries that are 6 months old but are still anonymous or do not have a paper associated with them. For conferences, 6 month is enough to determine if a paper has been accepted and to add the bibliography information. For longer review cycles, you need to resubmit your results.

Additional information used by the methods

Stereo: Method uses left and right (stereo) images
Flow: Method uses optical flow (2 temporally adjacent images)
Multiview: Method uses more than 2 temporally adjacent images
Laser Points: Method uses point clouds from Velodyne laser scanner
Additional training data: Use of additional data sources for training (see details)

Car

	Method	Setting	Code	Moderate	Easy	Hard	Runtime	Environment
1	Fusion-RCNN			88.35 %	89.59 %	79.56 %	0.1 s	1 core @ 2.5 Ghz (C/C++)

2	STD			87.76 %	89.66 %	86.89 %	0.08 s	GPU @ 2.5 Ghz (Python + C/C++)

3	Alibaba-AILabsX			87.49 %	89.10 %	78.98 %	0.05 s	1 core @ >3.5 Ghz (C/C++)

4	UberATG-MMF			87.47 %	89.49 %	79.10 %	0.08 s	GPU @ 2.5 Ghz (Python)
M. Liang, B. Yang, Y. Chen, R. Hu and R. Urtasun: Multi-Task Multi-Sensor Fusion for 3D Object Detection. CVPR 2019.
5	Patches - EMP			87.45 %	89.96 %	83.63 %	0.5 s	GPU @ 2.5 Ghz (Python)

6	GPOD			87.39 %	89.54 %	81.65 %	0.1 s	GPU @ 2.5 Ghz (Python)

7	Part-A^2			87.04 %	89.74 %	79.52 %	0.08 s	GPU @ 2.5 Ghz (Python + C/C++)

8	A-VoxelNet			86.96 %	88.30 %	78.44 %	0.029 s	GPU @ 2.5 Ghz (Python)

9	MPNet			86.95 %	89.28 %	80.32 %	0.02 s	GPU @ 2.5 Ghz (Python + C/C++)

10	MMV			86.90 %	88.61 %	78.45 %	0.4 s	GPU @ 2.5 Ghz (C/C++)

11	MODet			86.80 %	89.15 %	79.62 %	0.05 s	GTX1080Ti

12	SRF			86.75 %	88.60 %	78.89 %	0.05 s	GPU @ 2.5 Ghz (Python + C/C++)

13	Alibaba-AILabsX			86.71 %	88.84 %	78.62 %	0.2 s	GPU @ >3.5 Ghz (Python)

14	ARPNET			86.67 %	88.10 %	77.94 %	0.08 s	GPU @ 2.5 Ghz (Python + C/C++)

15	SECOND-V1.5		code	86.60 %	88.72 %	78.51 %	0.04 s	GPU @ 2.0 Ghz (Python + C/C++)

16	UberATG-HDNET			86.57 %	89.14 %	78.32 %	0.05 s	GPU @ 2.5 Ghz (Python)
B. Yang, M. Liang and R. Urtasun: HDNET: Exploiting HD Maps for 3D Object Detection. 2nd Conference on Robot Learning (CoRL) 2018.
17	Patches			86.55 %	89.78 %	79.22 %	0.15 s	GPU @ 2.0 Ghz

18	RAL			86.39 %	88.16 %	78.66 %	0.08 s	GPU @ 2.5 Ghz (Python + C/C++)

19	Tencent_ADlab_Lidar			86.29 %	88.67 %	79.27 %	0.1 s	GPU @ 2.5 Ghz (Python + C/C++)

20	TBA			86.29 %	88.75 %	80.13 %	0.07 s	1 core @ 2.5 Ghz (Python)

21	NU-optim			86.21 %	88.39 %	78.34 %	0.04 s	GPU @ >3.5 Ghz (Python)

22	SegVoxelNet			86.16 %	88.62 %	78.68 %	0.04 s	1 core @ 2.5 Ghz (Python)

23	PC-CNN-V2			86.10 %	88.49 %	77.26 %	0.5 s	GPU @ 2.5 Ghz (Matlab + C/C++)
X. Du, M. Ang, S. Karaman and D. Rus: A General Pipeline for 3D Detection of Vehicles. 2018 IEEE International Conference on Robotics and Automation (ICRA) 2018.
24	PointPillars		code	86.10 %	88.35 %	79.83 %	16 ms	1080ti GPU and Intel i7 CPU
A. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang and O. Beijbom: PointPillars: Fast Encoders for Object Detection from Point Clouds. CVPR 2019.
25	Fast Point R-CNNv1.1			86.10 %	88.03 %	78.17 %	0.06 s	GPU @ 2.5 Ghz (Python + C/C++)

26	DH-ARI			86.08 %	87.93 %	78.22 %	0.2 s	1 core @ >3.5 Ghz (Python + C/C++)

27	epBRM			86.08 %	88.75 %	78.80 %	0.1 s	GPU @ >3.5 Ghz (Python + C/C++)

28	PointRCNN-deprecated			86.04 %	89.28 %	79.02 %	0.1 s	GPU @ 2.5 Ghz (Python + C/C++)

29	MVX-Net			85.89 %	89.15 %	78.07 %	0.06 s	GPU @ 3.0 Ghz (Python + C/C++)

30	UberATG-ContFuse			85.83 %	88.81 %	77.33 %	0.06 s	GPU @ 2.5 Ghz (Python)
M. Liang, B. Yang, S. Wang and R. Urtasun: Deep Continuous Fusion for Multi-Sensor 3D Object Detection. ECCV 2018.
31	MDC			85.68 %	88.65 %	77.03 %	0.17 s	GPU @ 2.5 Ghz (Python)

32	PointRCNN		code	85.68 %	89.47 %	79.10 %	0.1 s	GPU @ 2.5 Ghz (Python + C/C++)
S. Shi, X. Wang and H. Li: PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud. CVPR 2019.
33	Fast Point R-CNN			85.53 %	87.98 %	77.68 %	0.06 s	GPU @ 2.5 Ghz (Python + C/C++)

34	AVOD		code	85.44 %	86.80 %	77.73 %	0.08 s	Titan X (pascal)
J. Ku, M. Mozifian, J. Lee, A. Harakeh and S. Waslander: Joint 3D Proposal Generation and Object Detection from View Aggregation. IROS 2018.
35	SeoulRobotics-HFD			85.22 %	87.17 %	77.71 %	0.035 s	GPU (C++)

36	Roadstar.ai			85.04 %	89.95 %	78.75 %	0.08 s	GPU @ 2.0 Ghz (Python)

37	CONV-BOX			84.56 %	87.54 %	77.79 %	0.2 s	Tesla V100

38	F-PointNet		code	84.00 %	88.70 %	75.33 %	0.17 s	GPU @ 3.0 Ghz (Python)
C. Qi, W. Liu, C. Wu, H. Su and L. Guibas: Frustum PointNets for 3D Object Detection from RGB-D Data. arXiv preprint arXiv:1711.08488 2017.
39	IPOD			83.98 %	86.93 %	77.85 %	0.2 s	GPU @ 2.5 Ghz (Python + C/C++)

40	AVOD-FPN		code	83.79 %	88.53 %	77.90 %	0.1 s	Titan X (Pascal)
J. Ku, M. Mozifian, J. Lee, A. Harakeh and S. Waslander: Joint 3D Proposal Generation and Object Detection from View Aggregation. IROS 2018.
41	UberATG-PIXOR++			83.70 %	89.38 %	77.97 %	0.035 s	GPU @ 2.5 Ghz (Python)
B. Yang, M. Liang and R. Urtasun: HDNET: Exploiting HD Maps for 3D Object Detection. 2nd Conference on Robot Learning (CoRL) 2018.
42	F-ConvNet			83.08 %	89.69 %	74.56 %	0.47 s	GPU @ 2.5 Ghz (Python + C/C++)
Z. Wang and K. Jia: Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal 3D Object Detection. arXiv preprint arXiv:1903.01864 2019.
43	PP_v1.0		code	81.12 %	87.72 %	78.13 %	0.02s	1 core @ 2.5 Ghz (C/C++)

44	PAD			80.51 %	87.87 %	78.36 %	0.15 s	1 core @ 2.5 Ghz (Python)

45	Voxel-FPN			80.17 %	88.96 %	79.14 %	0.02 s	GPU @ 2.5 Ghz (Python + C/C++)

46	PFPN			80.14 %	89.05 %	79.02 %	0.02 s	4 cores @ >3.5 Ghz (Python)

47	yl_net			80.06 %	85.64 %	77.52 %	0.03 s	GPU @ 2.5 Ghz (Python)

48	tiny_rfdet		code	80.05 %	86.53 %	76.28 %	0.01 s	GPU @ 2.5 Ghz (Python)

49	MVSLN			79.71 %	89.08 %	78.26 %	0.1s s	1 core @ 2.5 Ghz (C/C++)

50	CFR			79.68 %	88.77 %	78.21 %	0.06 s	1 core @ 2.5 Ghz (C/C++)

51	3DBN			79.40 %	88.13 %	77.97 %	0.13s	1080Ti (Python+C/C++)
X. Li, J. Guivant, N. Kwok and Y. Xu: 3D Backbone Network for 3D Object Detection. CoRR 2019.
52	SECOND		code	79.37 %	88.07 %	77.95 %	38 ms	1080Ti
Y. Yan, Y. Mao and B. Li: SECOND: Sparsely Embedded Convolutional Detection. Sensors 2018.
53	ELLIOT			79.30 %	87.06 %	77.28 %	0.1 s	1 core @ 2.5 Ghz (C/C++)

54	AILabs3D			79.09 %	88.32 %	77.40 %	0.6 s	GPU @ >3.5 Ghz (Python)

55	SCANet			79.01 %	87.02 %	77.20 %	0.17 s	>8 cores @ 2.5 Ghz (Python)

56	DFD			78.92 %	86.98 %	77.22 %	0.05 s	GPU @ 2.0 Ghz (Python + C/C++)

57	SCANet			78.64 %	87.36 %	77.37 %	0.09s	GPU @ 2.5 Ghz (Python)

58	SECA			78.54 %	87.52 %	77.25 %	0.09 s	GPU @ 2.5 Ghz (Python)

59	FNV1_RPN			78.51 %	87.40 %	70.23 %	0.12 s	1 core @ 2.5 Ghz (Python + C/C++)

60	SECA			78.50 %	87.65 %	76.99 %	1 s	GPU @ 2.5 Ghz (Python)

61	VSE			78.50 %	87.65 %	76.99 %	0.15 s	GPU @ 2.5 Ghz (Python)

62	NLK			78.28 %	87.65 %	76.20 %	0.02 s	1 core @ 2.5 Ghz (Python)

63	FNV1_Fusion			78.23 %	87.07 %	76.72 %	0.11 s	GPU @ 2.5 Ghz (Python)

64	X_MD			77.80 %	87.02 %	69.66 %	0.2 s	1 core @ 2.5 Ghz (Python + C/C++)

65	RTL3D			77.75 %	86.95 %	76.08 %	0.02 s	GPU @ 2.5 Ghz (Python)

66	FNV1			77.69 %	85.53 %	69.46 %	0.11 s	GPU @ 2.5 Ghz (Python)

67	VoxelNet(Unofficial)			77.58 %	86.08 %	69.33 %	0.5 s	GPU @ 2.0 Ghz (Python)

68	UberATG-PIXOR			77.05 %	81.70 %	72.95 %	0.035 s	TITAN Xp (Python)
B. Yang, W. Luo and R. Urtasun: PIXOR: Real-time 3D Object Detection from Point Clouds. CVPR 2018.
69	MV3D (LIDAR)			77.00 %	85.82 %	68.94 %	0.24 s	GPU @ 2.5 Ghz (Python + C/C++)
X. Chen, H. Ma, J. Wan, B. Li and T. Xia: Multi-View 3D Object Detection Network for Autonomous Driving. CVPR 2017.
70	MV3D			76.90 %	86.02 %	68.49 %	0.36 s	GPU @ 2.5 Ghz (Python + C/C++)
X. Chen, H. Ma, J. Wan, B. Li and T. Xia: Multi-View 3D Object Detection Network for Autonomous Driving. CVPR 2017.
71	Multi-3D			76.65 %	85.72 %	66.79 %	0.15 s	1 core @ 2.5 Ghz (C/C++)

72	FNV2			74.36 %	80.68 %	65.88 %	0.18 s	GPU @ 2.5 Ghz (Python)

73	LaserNet			73.77 %	78.25 %	66.47 %	12 ms	GPU @ 2.5 Ghz (C/C++)
G. Meyer, A. Laddha, E. Kee, C. Vallespi-Gonzalez and C. Wellington: LaserNet: An Efficient Probabilistic 3D Object Detector for Autonomous Driving. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2019.
74	CLF3D			73.01 %	80.09 %	58.56 %	0.13 s	GPU @ 2.5 Ghz (Python)

75	A3DODWTDA		code	72.86 %	76.65 %	64.51 %	0.08 s	GPU @ 3.0 Ghz (Python)
F. Gustafsson and E. Linder-Norén: Automotive 3D Object Detection Without Target Domain Annotations. 2018.
76	anm			72.73 %	81.79 %	63.98 %	3 s	1 core @ 2.5 Ghz (C/C++)

77	avodC			69.53 %	84.54 %	67.98 %	0.1 s	GPU @ 2.5 Ghz (Python)

78	TopNet-Retina			67.14 %	76.62 %	65.92 %	52ms	GeForce 1080Ti (tensorflow-gpu, v1.12)
S. Wirges, T. Fischer, C. Stiller and J. Frias: Object Detection and Classification in Occupancy Grid Maps Using Deep Convolutional Networks. 2018 21st International Conference on Intelligent Transportation Systems (ITSC) 2018.
79	Complexer-YOLO			66.07 %	74.23 %	65.70 %	0.06 s	GPU @ 3.5 Ghz (C/C++)

80	TopNet-DecayRate			64.12 %	79.76 %	56.48 %	92 ms	NVIDIA GeForce 1080 Ti (tensorflow-gpu)
S. Wirges, T. Fischer, C. Stiller and J. Frias: Object Detection and Classification in Occupancy Grid Maps Using Deep Convolutional Networks. 2018 21st International Conference on Intelligent Transportation Systems (ITSC) 2018.
81	3D FCN			62.54 %	69.94 %	55.94 %	>5 s	1 core @ 2.5 Ghz (C/C++)
B. Li: 3D Fully Convolutional Network for Vehicle Detection in Point Cloud. IROS 2017.
82	TopNet-UncEst			62.42 %	70.13 %	55.20 %	0.09 s	NVIDIA GeForce 1080 Ti (tensorflow-gpu)
S. Wirges, M. Braun, M. Lauer and C. Stiller: Capturing Object Detection Uncertainty in Multi-Layer Grid Maps. 2019.
83	Pseudo-LiDAR V2			57.49 %	75.97 %	53.61 %	0.4 s	GPU @ 2.5 Ghz (Python)

84	TopNet-HighRes			53.71 %	67.53 %	46.54 %	101ms	NVIDIA GeForce 1080 Ti (tensorflow-gpu)
S. Wirges, T. Fischer, C. Stiller and J. Frias: Object Detection and Classification in Occupancy Grid Maps Using Deep Convolutional Networks. 2018 21st International Conference on Intelligent Transportation Systems (ITSC) 2018.
85	BirdNet			50.81 %	75.52 %	50.00 %	0.11 s	Titan Xp GPU
J. Beltrán, C. Guindel, F. Moreno, D. Cruzado, F. García and A. Escalera: BirdNet: A 3D Object Detection Framework from LiDAR Information. 2018 21st International Conference on Intelligent Transportation Systems (ITSC) 2018.
86	RT3DStereo			49.48 %	59.32 %	43.16 %	0.08 s	GPU @ 2.5 Ghz (C/C++)

87	Pseudo-LiDAR		code	47.20 %	66.83 %	40.30 %	0.4 s	GPU @ 2.5 Ghz (C/C++)
Y. Wang, W. Chao, D. Garg, B. Hariharan, M. Campbell and K. Weinberger: Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving. CVPR 2019.
88	SA_3D			47.00 %	58.98 %	39.47 %	0.3 s	GPU @ 2.5 Ghz (Python)

89				45.11 %	65.08 %	38.42 %

90	Stereo R-CNN		code	43.87 %	61.67 %	36.44 %	0.3 s	GPU @ 2.5 Ghz (Python)
P. Li, X. Chen and S. Shen: Stereo R-CNN based 3D Object Detection for Autonomous Driving. CVPR 2019.
91	RT3D			42.10 %	54.68 %	44.05 %	0.09 s	GPU @ 1.8Ghz
Y. Zeng, Y. Hu, S. Liu, J. Ye, Y. Han, X. Li and N. Sun: RT3D: Real-Time 3-D Vehicle Detection in LiDAR Point Cloud for Autonomous Driving. IEEE Robotics and Automation Letters 2018.
92	Licar			40.40 %	45.81 %	37.09 %	0.09 s	GPU @ 2.0 Ghz (Python)

93	DLMB			37.10 %	45.12 %	32.88 %	0.03 s	8 cores @ 3.5 Ghz (C/C++)

94	StereoFENet			34.62 %	49.14 %	28.41 %	0.15 s	1 core @ 3.5 Ghz (Python)
W. Bao, B. Xu and Z. Chen: MonoFENet: Monocular 3D Object Detection with Feature Enhancement Networks. 2019.
95	monocular			22.24 %	27.91 %	18.62 %	0.4 s	GPU @ 2.5 Ghz (Python + C/C++)

96	MonoDIS			19.08 %	18.88 %	17.41 %	0.1 s	1 core @ 2.5 Ghz (C/C++)

97	CSoR			18.69 %	23.94 %	16.30 %	3.5 s	4 cores @ >3.5 Ghz (Python + C/C++)
L. Plotkin: PyDriver: Entwicklung eines Frameworks für räumliche Detektion und Klassifikation von Objekten in Fahrzeugumgebung. 2015.
98	MonoPSR			17.66 %	20.25 %	15.78 %	0.2 s	GPU @ 3.5 Ghz (Python)
J. Ku, A. Pon and S. Waslander: Monocular 3D Object Detection Leveraging Accurate Proposals and Shape Reconstruction. CVPR 2019.
99	DT3D			17.19 %	23.38 %	13.86 %	0,21s	GPU @ 2.5 Ghz (Python)

100	MonoGRNet		code	16.37 %	20.55 %	15.16 %	0.04s	NVIDIA P40
Z. Qin, J. Wang and Y. Lu: MonoGRNet: A Geometric Reasoning Network for 3D Object Localization. The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19) 2019.
101	mylsi-faster-rcnn			15.99 %	21.09 %	14.56 %	0.3 s	1 core @ 2.5 Ghz (Python)

102	SS3D			12.81 %	17.86 %	12.28 %	48 ms	Tesla V100 (Python)

103	MonoFENet			12.71 %	18.08 %	10.55 %	0.15 s	1 core @ 3.5 Ghz (Python)
W. Bao, B. Xu and Z. Chen: MonoFENet: Monocular 3D Object Detection with Feature Enhancement Networks. 2019.
104	ROI-10D			12.40 %	16.77 %	11.39 %	0.2 s	GPU @ 3.5 Ghz (Python)
F. Manhardt, W. Kehl and A. Gaidon: ROI-10D: Monocular Lifting of 2D Detection to 6D Pose and Metric Shape. Computer Vision and Pattern Recognition (CVPR) 2019.
105	mymask-rcnn			12.17 %	19.13 %	10.56 %	0.3 s	1 core @ 2.5 Ghz (Python)

106	A3DODWTDA (image)		code	10.21 %	10.61 %	8.64 %	0.8 s	GPU @ 3.0 Ghz (Python)
F. Gustafsson and E. Linder-Norén: Automotive 3D Object Detection Without Target Domain Annotations. 2018.
107	GS3D			9.12 %	11.30 %	7.23 %	2 s	1 core @ 2.5 Ghz (C/C++)
B. Li, W. Ouyang, L. Sheng, X. Zeng and X. Wang: GS3D: An Efficient 3D Object Detection Framework for Autonomous Driving. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2019.
108	Shift R-CNN			8.49 %	13.32 %	6.40 %	0.25 s	GPU @ 1.5 Ghz (Python)
A. Naiden, V. Paunescu, G. Kim, B. Jeon and M. Leordeanu: Shift R-CNN: Deep Monocular 3D Object Detection With Closed-form Geometric Constraints. ICIP 2019.
109	OFT-Net			7.99 %	9.50 %	7.51 %	0.5 s	8 cores @ 2.5 Ghz (Python + C/C++)

110	RAR-Net			6.21 %	10.38 %	5.43 %	0.5 s	1 core @ 2.5 Ghz (C/C++)

111	MF3D			5.57 %	7.88 %	5.08 %	0.03 s	GPU @ 2.5 Ghz (C/C++)

112	FQNet			4.62 %	6.51 %	3.99 %	0.5 s	1 core @ 2.5 Ghz (Python)

113	monoref3d			3.46 %	4.74 %	2.68 %	0.1 s	1 core @ 2.5 Ghz (Python)

114	ref3D			3.46 %	4.74 %	2.68 %	0.1 s	1 core @ 2.5 Ghz (Python + C/C++)

115	3D-SSMFCNN		code	3.19 %	3.66 %	3.45 %	0.1 s	GPU @ 1.5 Ghz (C/C++)
L. Novak: Vehicle Detection and Pose Estimation for Autonomous Driving. 2017.
116	3DVSSD			2.01 %	2.02 %	1.68 %	0.06 s	1 core @ 2.5 Ghz (C/C++)

117	VeloFCN			0.33 %	0.15 %	0.47 %	1 s	GPU @ 2.5 Ghz (Python + C/C++)
B. Li, T. Zhang and T. Xia: Vehicle Detection from 3D Lidar Using Fully Convolutional Network. RSS 2016 .
118	ref3D			0.00 %	0.00 %	0.01 %	0.1 s	1 core @ 2.5 Ghz (Python)

119	multi-task CNN			0.00 %	0.00 %	0.00 %	25.1 ms	GPU @ 2.0 Ghz (Python)
M. Oeljeklaus, F. Hoffmann and T. Bertram: A Fast Multi-Task CNN for Spatial Understanding of Traffic Scenes. IEEE Intelligent Transportation Systems Conference 2018.
120	mBoW			0.00 %	0.00 %	0.00 %	10 s	1 core @ 2.5 Ghz (C/C++)
J. Behley, V. Steinhage and A. Cremers: Laser-based Segment Classification Using a Mixture of Bag-of-Words. Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2013.

Table as LaTeX | Only published Methods

Pedestrian

	Method	Setting	Code	Moderate	Easy	Hard	Runtime	Environment
1	A-VoxelNet			51.90 %	60.44 %	49.48 %	0.029 s	GPU @ 2.5 Ghz (Python)

2	VMVS			51.73 %	61.46 %	47.69 %	0.25 s	GPU @ 2.5 Ghz (Python)

3	STD			51.39 %	60.99 %	45.89 %	0.08 s	GPU @ 2.5 Ghz (Python + C/C++)

4	IPOD			51.24 %	60.83 %	45.40 %	0.2 s	GPU @ 2.5 Ghz (Python + C/C++)

5	AVOD-FPN		code	51.05 %	58.75 %	47.54 %	0.1 s	Titan X (Pascal)
J. Ku, M. Mozifian, J. Lee, A. Harakeh and S. Waslander: Joint 3D Proposal Generation and Object Detection from View Aggregation. IROS 2018.
6	F-ConvNet			50.48 %	58.90 %	46.72 %	0.47 s	GPU @ 2.5 Ghz (Python + C/C++)
Z. Wang and K. Jia: Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal 3D Object Detection. arXiv preprint arXiv:1903.01864 2019.
7	PointPillars		code	50.23 %	58.66 %	47.19 %	16 ms	1080ti GPU and Intel i7 CPU
A. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang and O. Beijbom: PointPillars: Fast Encoders for Object Detection from Point Clouds. CVPR 2019.
8	F-PointNet		code	50.22 %	58.09 %	47.20 %	0.17 s	GPU @ 3.0 Ghz (Python)
C. Qi, W. Liu, C. Wu, H. Su and L. Guibas: Frustum PointNets for 3D Object Detection from RGB-D Data. arXiv preprint arXiv:1711.08488 2017.
9	Tencent_ADlab_Lidar			48.93 %	56.89 %	46.28 %	0.1 s	GPU @ 2.5 Ghz (Python + C/C++)

10	Multi-3D			47.78 %	54.91 %	43.34 %	0.15 s	1 core @ 2.5 Ghz (C/C++)

11	PointRCNN		code	47.53 %	55.92 %	44.67 %	0.1 s	GPU @ 2.5 Ghz (Python + C/C++)
S. Shi, X. Wang and H. Li: PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud. CVPR 2019.
12	epBRM			47.17 %	53.24 %	44.83 %	0.10 s	1 core @ 2.5 Ghz (C/C++)

13	PP_v1.0		code	46.48 %	54.26 %	44.14 %	0.02s	1 core @ 2.5 Ghz (C/C++)

14	GPOD			46.39 %	53.09 %	43.62 %	0.1 s	GPU @ 2.5 Ghz (Python)

15	SECOND		code	46.27 %	55.10 %	44.76 %	38 ms	1080Ti
Y. Yan, Y. Mao and B. Li: SECOND: Sparsely Embedded Convolutional Detection. Sensors 2018.
16	Roadstar.ai			46.22 %	49.94 %	44.14 %	0.08 s	GPU @ 2.0 Ghz (Python)

17	CONV-BOX			45.09 %	52.71 %	43.90 %	0.2 s	Tesla V100

18	MDC			45.02 %	53.51 %	43.67 %	0.17 s	GPU @ 2.5 Ghz (Python)

19	SCANet			44.30 %	53.68 %	42.65 %	0.17 s	>8 cores @ 2.5 Ghz (Python)

20	ARPNET			43.35 %	50.80 %	37.79 %	0.08 s	GPU @ 2.5 Ghz (Python + C/C++)

21	ELLIOT			42.66 %	50.68 %	39.67 %	0.1 s	1 core @ 2.5 Ghz (C/C++)

22	anm			40.48 %	50.07 %	35.64 %	3 s	1 core @ 2.5 Ghz (C/C++)

23	CFR			39.24 %	52.47 %	38.07 %	0.06 s	1 core @ 2.5 Ghz (C/C++)

24	SA_3D			35.93 %	47.45 %	34.99 %	0.3 s	GPU @ 2.5 Ghz (Python)

25	anonymous			35.58 %	42.50 %	35.06 %	0.75 s	GPU @ 3.5 Ghz (C/C++)

26	X_MD			35.57 %	43.28 %	34.25 %	0.2 s	1 core @ 2.5 Ghz (C/C++)

27				35.26 %	43.26 %	32.85 %

28	AVOD		code	35.24 %	42.51 %	33.97 %	0.08 s	Titan X (pascal)
J. Ku, M. Mozifian, J. Lee, A. Harakeh and S. Waslander: Joint 3D Proposal Generation and Object Detection from View Aggregation. IROS 2018.
29	CLF3D			34.25 %	43.11 %	33.01 %	0.13 s	GPU @ 2.5 Ghz (Python)

30	BirdNet			21.35 %	26.07 %	19.96 %	0.11 s	Titan Xp GPU
J. Beltrán, C. Guindel, F. Moreno, D. Cruzado, F. García and A. Escalera: BirdNet: A 3D Object Detection Framework from LiDAR Information. 2018 21st International Conference on Intelligent Transportation Systems (ITSC) 2018.
31	Complexer-YOLO			20.88 %	22.00 %	20.81 %	0.06 s	GPU @ 3.5 Ghz (C/C++)

32	TopNet-HighRes			19.08 %	24.30 %	18.46 %	101ms	NVIDIA GeForce 1080 Ti (tensorflow-gpu)
S. Wirges, T. Fischer, C. Stiller and J. Frias: Object Detection and Classification in Occupancy Grid Maps Using Deep Convolutional Networks. 2018 21st International Conference on Intelligent Transportation Systems (ITSC) 2018.
33	TopNet-Retina			17.62 %	18.28 %	14.08 %	52ms	GeForce 1080Ti (tensorflow-gpu, v1.12)
S. Wirges, T. Fischer, C. Stiller and J. Frias: Object Detection and Classification in Occupancy Grid Maps Using Deep Convolutional Networks. 2018 21st International Conference on Intelligent Transportation Systems (ITSC) 2018.
34	TopNet-DecayRate			12.59 %	15.09 %	12.23 %	92 ms	NVIDIA GeForce 1080 Ti (tensorflow-gpu)
S. Wirges, T. Fischer, C. Stiller and J. Frias: Object Detection and Classification in Occupancy Grid Maps Using Deep Convolutional Networks. 2018 21st International Conference on Intelligent Transportation Systems (ITSC) 2018.
35	TopNet-UncEst			11.63 %	12.39 %	11.39 %	0.09 s	NVIDIA GeForce 1080 Ti (tensorflow-gpu)
S. Wirges, M. Braun, M. Lauer and C. Stiller: Capturing Object Detection Uncertainty in Multi-Layer Grid Maps. 2019.
36	Shift R-CNN			11.44 %	13.81 %	10.76 %	0.25 s	GPU @ 1.5 Ghz (Python)
A. Naiden, V. Paunescu, G. Kim, B. Jeon and M. Leordeanu: Shift R-CNN: Deep Monocular 3D Object Detection With Closed-form Geometric Constraints. ICIP 2019.
37	MonoPSR			11.22 %	14.27 %	10.54 %	0.2 s	GPU @ 3.5 Ghz (Python)
J. Ku, A. Pon and S. Waslander: Monocular 3D Object Detection Leveraging Accurate Proposals and Shape Reconstruction. CVPR 2019.
38	RT3DStereo			5.30 %	5.39 %	5.19 %	0.08 s	GPU @ 2.5 Ghz (C/C++)

39	SS3D			3.52 %	3.86 %	2.50 %	48 ms	Tesla V100 (Python)

40	mylsi-faster-rcnn			3.52 %	4.56 %	3.52 %	0.3 s	1 core @ 2.5 Ghz (Python)

41	mymask-rcnn			2.64 %	3.91 %	2.63 %	0.3 s	1 core @ 2.5 Ghz (Python)

42	OFT-Net			1.55 %	1.93 %	1.65 %	0.5 s	8 cores @ 2.5 Ghz (Python + C/C++)

43	DT3D			1.22 %	1.15 %	1.14 %	0,21s	GPU @ 2.5 Ghz (Python)

44	mBoW			0.01 %	0.01 %	0.01 %	10 s	1 core @ 2.5 Ghz (C/C++)
J. Behley, V. Steinhage and A. Cremers: Laser-based Segment Classification Using a Mixture of Bag-of-Words. Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2013.

Table as LaTeX | Only published Methods

Cyclist

	Method	Setting	Code	Moderate	Easy	Hard	Runtime	Environment
1	F-ConvNet			68.62 %	82.59 %	60.62 %	0.47 s	GPU @ 2.5 Ghz (Python + C/C++)
Z. Wang and K. Jia: Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal 3D Object Detection. arXiv preprint arXiv:1903.01864 2019.
2	PointRCNN		code	66.77 %	81.52 %	60.78 %	0.1 s	GPU @ 2.5 Ghz (Python + C/C++)
S. Shi, X. Wang and H. Li: PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud. CVPR 2019.
3	STD			65.32 %	81.04 %	57.85 %	0.08 s	GPU @ 2.5 Ghz (Python + C/C++)

4	MDC			64.78 %	77.38 %	56.77 %	0.17 s	GPU @ 2.5 Ghz (Python)

5	Multi-3D			63.72 %	78.76 %	55.50 %	0.15 s	1 core @ 2.5 Ghz (C/C++)

6	ARPNET			62.76 %	77.99 %	55.55 %	0.08 s	GPU @ 2.5 Ghz (Python + C/C++)

7	Tencent_ADlab_Lidar			62.33 %	78.15 %	55.87 %	0.1 s	GPU @ 2.5 Ghz (Python + C/C++)

8	PointPillars		code	62.25 %	79.14 %	56.00 %	16 ms	1080ti GPU and Intel i7 CPU
A. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang and O. Beijbom: PointPillars: Fast Encoders for Object Detection from Point Clouds. CVPR 2019.
9	F-PointNet		code	61.96 %	75.38 %	54.68 %	0.17 s	GPU @ 3.0 Ghz (Python)
C. Qi, W. Liu, C. Wu, H. Su and L. Guibas: Frustum PointNets for 3D Object Detection from RGB-D Data. arXiv preprint arXiv:1711.08488 2017.
10	CONV-BOX			61.84 %	71.60 %	55.03 %	0.2 s	Tesla V100

11	A-VoxelNet			60.85 %	76.71 %	54.27 %	0.029 s	GPU @ 2.5 Ghz (Python)

12	GPOD			60.73 %	68.84 %	55.05 %	0.1 s	GPU @ 2.5 Ghz (Python)

13	Roadstar.ai			60.26 %	67.31 %	54.62 %	0.08 s	GPU @ 2.0 Ghz (Python)

14	epBRM			60.22 %	73.33 %	54.10 %	0.10 s	1 core @ 2.5 Ghz (C/C++)

15	SCANet			59.09 %	70.73 %	52.79 %	0.17 s	>8 cores @ 2.5 Ghz (Python)

16	IPOD			58.92 %	77.10 %	51.01 %	0.2 s	GPU @ 2.5 Ghz (Python + C/C++)

17	ELLIOT			57.63 %	76.27 %	52.36 %	0.1 s	1 core @ 2.5 Ghz (C/C++)

18	AVOD-FPN		code	57.48 %	68.09 %	50.77 %	0.1 s	Titan X (Pascal)
J. Ku, M. Mozifian, J. Lee, A. Harakeh and S. Waslander: Joint 3D Proposal Generation and Object Detection from View Aggregation. IROS 2018.
19	SECOND		code	56.04 %	73.67 %	48.78 %	38 ms	1080Ti
Y. Yan, Y. Mao and B. Li: SECOND: Sparsely Embedded Convolutional Detection. Sensors 2018.
20	PP_v1.0		code	54.59 %	70.83 %	49.11 %	0.02s	1 core @ 2.5 Ghz (C/C++)

21	CFR			53.41 %	69.05 %	46.81 %	0.06 s	1 core @ 2.5 Ghz (C/C++)

22	AVOD		code	47.74 %	63.66 %	46.55 %	0.08 s	Titan X (pascal)
J. Ku, M. Mozifian, J. Lee, A. Harakeh and S. Waslander: Joint 3D Proposal Generation and Object Detection from View Aggregation. IROS 2018.
23	X_MD			44.49 %	54.22 %	38.48 %	0.2 s	1 core @ 2.5 Ghz (C/C++)

24	CLF3D			41.57 %	53.29 %	35.80 %	0.13 s	GPU @ 2.5 Ghz (Python)

25	anm			40.42 %	56.84 %	35.20 %	3 s	1 core @ 2.5 Ghz (C/C++)

26	TopNet-Retina			38.11 %	50.15 %	36.49 %	52ms	GeForce 1080Ti (tensorflow-gpu, v1.12)
S. Wirges, T. Fischer, C. Stiller and J. Frias: Object Detection and Classification in Occupancy Grid Maps Using Deep Convolutional Networks. 2018 21st International Conference on Intelligent Transportation Systems (ITSC) 2018.
27	Complexer-YOLO			30.16 %	36.12 %	26.01 %	0.06 s	GPU @ 3.5 Ghz (C/C++)

28	BirdNet			27.18 %	38.93 %	25.51 %	0.11 s	Titan Xp GPU
J. Beltrán, C. Guindel, F. Moreno, D. Cruzado, F. García and A. Escalera: BirdNet: A 3D Object Detection Framework from LiDAR Information. 2018 21st International Conference on Intelligent Transportation Systems (ITSC) 2018.
29	SA_3D			20.56 %	26.45 %	16.42 %	0.3 s	GPU @ 2.5 Ghz (Python)

30	TopNet-DecayRate			19.92 %	28.06 %	19.13 %	92 ms	NVIDIA GeForce 1080 Ti (tensorflow-gpu)
S. Wirges, T. Fischer, C. Stiller and J. Frias: Object Detection and Classification in Occupancy Grid Maps Using Deep Convolutional Networks. 2018 21st International Conference on Intelligent Transportation Systems (ITSC) 2018.
31	TopNet-HighRes			12.45 %	15.70 %	12.76 %	101ms	NVIDIA GeForce 1080 Ti (tensorflow-gpu)
S. Wirges, T. Fischer, C. Stiller and J. Frias: Object Detection and Classification in Occupancy Grid Maps Using Deep Convolutional Networks. 2018 21st International Conference on Intelligent Transportation Systems (ITSC) 2018.
32	MonoPSR			12.17 %	14.75 %	11.35 %	0.2 s	GPU @ 3.5 Ghz (Python)
J. Ku, A. Pon and S. Waslander: Monocular 3D Object Detection Leveraging Accurate Proposals and Shape Reconstruction. CVPR 2019.
33	TopNet-UncEst			10.34 %	14.80 %	10.42 %	0.09 s	NVIDIA GeForce 1080 Ti (tensorflow-gpu)
S. Wirges, M. Braun, M. Lauer and C. Stiller: Capturing Object Detection Uncertainty in Multi-Layer Grid Maps. 2019.
34	SS3D			9.65 %	11.52 %	9.09 %	48 ms	Tesla V100 (Python)

35	RT3DStereo			7.59 %	7.70 %	7.51 %	0.08 s	GPU @ 2.5 Ghz (C/C++)

36	Shift R-CNN			3.03 %	3.58 %	3.03 %	0.25 s	GPU @ 1.5 Ghz (Python)
A. Naiden, V. Paunescu, G. Kim, B. Jeon and M. Leordeanu: Shift R-CNN: Deep Monocular 3D Object Detection With Closed-form Geometric Constraints. ICIP 2019.
37	mylsi-faster-rcnn			2.75 %	3.06 %	2.57 %	0.3 s	1 core @ 2.5 Ghz (Python)

38	DT3D			1.26 %	2.54 %	1.47 %	0,21s	GPU @ 2.5 Ghz (Python)

39	mymask-rcnn			1.12 %	2.20 %	1.18 %	0.3 s	1 core @ 2.5 Ghz (Python)

40	OFT-Net			0.43 %	0.79 %	0.43 %	0.5 s	8 cores @ 2.5 Ghz (Python + C/C++)

41	mBoW			0.00 %	0.00 %	0.00 %	10 s	1 core @ 2.5 Ghz (C/C++)
J. Behley, V. Steinhage and A. Cremers: Laser-based Segment Classification Using a Mixture of Bag-of-Words. Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2013.

Table as LaTeX | Only published Methods

Related Datasets

CERV Vehicle Lights Dataset: Annotations of vehicle lights for a subset of the object detection benchmark.
PASCAL3D+: Augments 12 rigid object classes of PASCAL VOC 2012 with 3D annotations.
The PASCAL Visual Object Classes Challenges: Dataset and benchmarks for object class recognition.
TME Motorway Dataset: 28 video sequences with vehicle annotations captured from VisLab's BRAiVE vehicle.
LabelMe: Online annotation tool to build image databases for computer vision research.
MIT Street Scenes: Street-side images with labels for 9 object categories (including cars, pedestrians, buildings, trees).
Daimler Pedestrian Datasets: Datasets focusing on pedestrian detection for autonomous driving.
Caltech Pedestrian Detection Benchmark: 10 hours of video with 350.000 annotated pedestrian bounding boxes.
Robust Multi-Person Tracking from Mobile Platforms: Videos with annotated pedestrians captured from a stroller.

Citation

When using this dataset in your research, we will be happy if you cite us:
@INPROCEEDINGS{Geiger2012CVPR,
author = {Andreas Geiger and Philip Lenz and Raquel Urtasun},
title = {Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite},
booktitle = {Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2012}
}

The KITTI Vision Benchmark Suite

A project of Karlsruhe Institute of Technologyand Toyota Technological Institute at Chicago

Bird's Eye View Evaluation 2017

Car

Pedestrian

Cyclist

Related Datasets

Citation

A project of Karlsruhe Institute of Technology
and Toyota Technological Institute at Chicago