LiDAR–Camera Object-Level Fusion for Multi-Target Tracking Using JPDA and EKF: A Reproducible Empirical Study on a PandaSet-Parameterised Five-Sequence Dataset

Qi Xin

doi:10.51903/jtie.v5i1.486

Authors

Qi Xin Management Information Systems, University of Pittsburgh, PA, USA

DOI:

https://doi.org/10.51903/jtie.v5i1.486

Keywords:

LiDAR–camera fusion, multi-target tracking, JPDA, extended Kalman filter, data association

Abstract

Multi-target tracking in cluttered scenes is essential for automated driving, where downstream planning requires stable object identities and accurate state estimates. This paper provides a fully reproducible empirical and sensitivity study of a classical object-level LiDAR–camera fusion tracker that combines Joint Probabilistic Data Association (JPDA) with an Extended Kalman Filter (EKF) under a constant-velocity state model. Because the MathWorks PandaSet subset is distributed as a ZIP archive that cannot be ingested into our execution environment, we generate a PandaSet-parameterised five-sequence synthetic dataset with explicitly specified sampling rates, measurement noise, detection probabilities, and Poisson clutter, and report end-to-end results with fixed random seeds. Using sequential fusion (LiDAR JPDA–EKF update followed by a camera bearing update), we obtain a mean MOTA of 0.880 and a mean position RMSE of 0.361 m, compared with LiDAR-only JPDA–EKF MOTA of 0.883 and RMSE of 0.395 m. Fusion, therefore, improves localization accuracy while sometimes reducing MOTA due to additional association ambiguity introduced by camera clutter; this trade-off is discussed in terms of downstream use cases that prioritize state accuracy. Sensitivity sweeps show that probabilistic association degrades more gracefully than hard nearest-neighbor assignment as clutter increases and delineate regimes where camera information is beneficial. A camera-only bearing tracker is included as a diagnostic baseline (not as a competitive approach); as expected, given the observability limits, it is not reliable under the studied clutter conditions. The dataset specification, parameters, and reporting artefacts form a reproducible template for diagnosing JPDA/EKF tracking and object-level fusion.

References

Bai, X., Hu, Z., Zhu, X., Huang, Q., Chen, Y., Fu, H., & Tai, C.-L. (2022). TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 1080–1089. https://doi.org/10.1109/cvpr52688.2022.00116

Bar-Shalom, Y., & Li, X. R. (1995). Multitarget-Multisensor Tracking: Principles and Techniques. YBS Publishing. https://books.google.com/books?id=ev7vAAAAMAAJ

Bar-Shalom, Y., & Tse, E. (1975). Tracking in a Cluttered Environment with Probabilistic Data Association. Automatica, 11(5), 451–460. https://doi.org/10.1016/0005 1098(75)90021 7

Bar Shalom, Y., Li, X. R., & Kirubarajan, T. (2001). Estimation with Applications to Tracking and Navigation: Theory, Algorithms and Software. John Wiley & Sons. https://openlibrary.org/works/OL16965810W/estimation_with_applications_to_tracking_and_navigation

Bernardin, K., & Stiefelhagen, R. (2008). Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics. EURASIP Journal on Image and Video Processing, 2008(1), 246309. https://doi.org/10.1155/2008/246309

Bewley, A., Ge, Z., Ott, L., Ramos, F., & Upcroft, B. (2016). Simple Online and Realtime Tracking. In 2016 IEEE International Conference on Image Processing (ICIP), 3464–3468. https://doi.org/10.1109/icip.2016.7533003

Blackman, S. S., & Popoli, R. (1999). Design and Analysis of Modern Tracking Systems. Artech House. https://openlibrary.org/works/OL15080957W/design_and_analysis_of_modern_tracking_systems

Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., & Beijbom, O. (2020). nuScenes: A Multimodal Dataset for Autonomous Driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 11618–11628. https://doi.org/10.1109/cvpr42600.2020.01164

Chang, M.-F., Lambert, J., Sangkloy, P., Singh, J., Bak, S., Hartnett, A., Wang, D., Carr, P., Lucey, S., Ramanan, D., Hays, J., & Savarese, S. (2019). Argoverse: 3D Tracking and Forecasting with Rich Maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 8748–8757. https://doi.org/10.1109/cvpr.2019.00895

Du, Y., Zhao, Z., Song, Y., Zhao, Y., Su, F., Gong, T., & Meng, H. (2023). StrongSORT: Make DeepSORT Great Again. IEEE Transactions on Multimedia. IEEE Transactions on Multimedia, 25, 8725-8737. https://doi.org/10.1109/tmm.2023.3240881

Fortmann, T. E., Bar-Shalom, Y., & Scheffe, M. (1983). Sonar Tracking of Multiple Targets Using Joint Probabilistic Data Association. IEEE Journal of Oceanic Engineering, 8(3), 173–184. https://doi.org/10.1109/joe.1983.1145560

Geiger, A., Lenz, P., & Urtasun, R. (2012). Are We Ready for Autonomous Driving? The KITTI Vision Benchmark Suite. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, 3354–3361. https://doi.org/10.1109/cvpr.2012.6248074

Hu, Y., Yang, J., Chen, L., Li, K., Sima, C., Zhu, X., et al. (2023). Planning Oriented Autonomous Driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 17853–17862. https://doi.org/10.1109/cvpr52729.2023.01712

Kalman, R. E. (1960). A New Approach to Linear Filtering and Prediction Problems. Journal of Basic Engineering, 82(1), 35–45. https://doi.org/10.1115/1.3662552

Kim, S. R., Park, J. H., & Hong, J. Y. (2025). A Hybrid Noise Reduction and Normalization Framework for Improving Multimodal Sensor Data Quality in Real-Time Systems. Journal of Technology Informatics and Engineering, 4(3), 350–368. https://doi.org/10.51903/jtie.v4i3.440

Liu, Z., Tang, H., Amini, A., Yang, X., Mao, H., Rus, D., & Han, S. (2023). BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird’s-Eye View Representation. In 2023 IEEE International Conference on Robotics and Automation (ICRA). https://doi.org/10.1109/icra48891.2023.10160968

Luiten, J., Osep, A., Dendorfer, P., Torr, P., Geiger, A., Leal-Taixé, L., & Leibe, B. (2021). HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking. International Journal of Computer Vision, 129, 548–578. https://doi.org/10.1007/s11263-020-01375-2

Mahler, R. P. S. (2007). Statistical Multisource-Multitarget Information Fusion. Artech House. https://openlibrary.org/works/OL15085988W/statistical_multisource-multitarget_information_fusion

MathWorks. (2026). trackCLEARMetrics (CLEAR Multi-Object Tracking Metrics) Documentation. https://www.mathworks.com/help/fusion/ref/trackclearmetrics.html

Meinhardt, T., Kirillov, A., Leal Taixé, L., & Feichtenhofer, C. (2022). TrackFormer: Multi Object Tracking with Transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 7363–7373. https://doi.org/10.1109/cvpr52688.2022.00725

Reid, D. B. (1979). An Algorithm for Tracking Multiple Targets. IEEE Transactions on Automatic Control, 24(6), 843–854. https://doi.org/10.1109/tac.1979.1102177

Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., Caine, B., Vasudevan, V., Han, W., Ngiam, J., Zhao, H., Timofeev, A., Ettinger, S., Krivokon, M., Gao, A., Joshi, T., Zhang, Y., Shlens, J., Chen, Z., & Anguelov, D. (2020). Scalability in Perception for Autonomous Driving: Waymo Open Dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2446–2454. https://doi.org/10.1109/cvpr42600.2020.00250

Vo, B.-N., & Ma, W.-K. (2006). The Gaussian Mixture Probability Hypothesis Density Filter. IEEE Transactions on Signal Processing, 54(11), 4091–4104. https://doi.org/10.1109/tsp.2006.881190

Welch, G., & Bishop, G. (2006). An Introduction to the Kalman Filter. University of North Carolina at Chapel Hill, Department of Computer Science. http://www.cs.unc.edu/~welch/media/pdf/kalman_intro.pdf

Wojke, N., Bewley, A., & Paulus, D. (2017). Simple Online And Realtime Tracking with a Deep Association Metric. In 2017 IEEE International Conference on Image Processing (ICIP), 3645–3649. https://doi.org/10.1109/icip.2017.8296962

Xiao, P., Shao, Z., Hao, Q., Zhang, L., Chai, Y., Li, X., Wu, Z., Sun, P., & Chen, Z. (2021). PandaSet: Advanced Sensor Suite Dataset for Autonomous Driving. In 2021 IEEE Intelligent Transportation Systems Conference (ITSC), 3098–3105. https://doi.org/10.1109/itsc48978.2021.9565009

Yan, F., Luo, W., Zhong, Y., Gan, Y., & Ma, L. (2025). Yan, F., Luo, W., Zhong, Y., Gan, Y., & Ma, L. (2025). Co MOT: Boosting End to End Transformer Based Multi Object Tracking via Coopetition Label Assignment and Shadow Sets. In International Conference on Learning Representations (ICLR) 2025. https://proceedings.iclr.cc/paper/2025/hash/8428da31da191712130ce8cce265691a Abstract Conference.html

Zhang, Y., Sun, P., Jiang, Y., Yu, D., Weng, F., Yuan, Z., et al. (2022). ByteTrack: Multi-Object Tracking by Associating Every Detection Box. In European Conference on Computer Vision (ECCV), 1-21. doi:10.1007/978-3-031-20047-2_1