A 3D Multi-Object Tracking Framework Based on Camera-LiDAR Fusion with Deep Association[st][la] [on] [DeepFusion-MOT]

Submitted on 19 Nov. 2021 04:50 by
Xiyang Wang (Chongqing University (CQU SLAMMOT Team))

Running time:0.01 s
Environment:>8 cores @ 2.5 Ghz (Python)

Method Description:
This paper proposes a robust and fast camera-LiDAR
fusion-based MOT method that achieves a good
trade-off between accuracy and speed. Relying on
the characteristics of camera and LiDAR sensors,
an effective deep association mechanism is
designed and embedded in the proposed MOT method.
This association mechanism realizes tracking of an
object in a 2D domain when the object is far away
and only detected by the camera, and updating of
the 2D trajectory with 3D information obtained
when the object appears in the LiDAR field of view
to achieve a smooth fusion of 2D and 3D
trajectories. Extensive experiments based on the
KITTI dataset indicate that our proposed method
presents obvious advantages over the state-of-the-
art MOT methods in terms of both tracking accuracy
and processing speed. Code
See the code for details.
Latex Bibtex:
@ARTICLE{9810346, author={Wang, Xiyang and Fu,
Chunyun and Li, Zhankun and Lai, Ying and He,
Jiawei}, journal={IEEE Robotics and Automation
Letters}, title={DeepFusionMOT: A 3D Multi-Object
Tracking Framework Based on Camera-LiDAR Fusion with
Deep Association}, year={2022}, volume={},
number={}, pages={1-8}, doi=

Detailed Results

From all 29 test sequences, our benchmark computes the HOTA tracking metrics (HOTA, DetA, AssA, DetRe, DetPr, AssRe, AssPr, LocA) [1] as well as the CLEARMOT, MT/PT/ML, identity switches, and fragmentation [2,3] metrics. The tables below show all of these metrics.

Benchmark HOTA DetA AssA DetRe DetPr AssRe AssPr LocA
CAR 75.46 % 71.54 % 80.05 % 75.34 % 85.25 % 82.63 % 89.77 % 86.70 %

Benchmark TP FP FN
CAR 29791 4601 601

CAR 84.63 % 85.02 % 84.87 % 84 71.66 %

Benchmark MT rate PT rate ML rate FRAG
CAR 68.61 % 22.31 % 9.08 % 472

Benchmark # Dets # Tracks
CAR 30392 968

This table as LaTeX

This figure as: png pdf

[1] J. Luiten, A. Os̆ep, P. Dendorfer, P. Torr, A. Geiger, L. Leal-Taixé, B. Leibe: HOTA: A Higher Order Metric for Evaluating Multi-object Tracking. IJCV 2020.
[2] K. Bernardin, R. Stiefelhagen: Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics. JIVP 2008.
[3] Y. Li, C. Huang, R. Nevatia: Learning to associate: HybridBoosted multi-target tracker for crowded scene. CVPR 2009.

eXTReMe Tracker