A Detachable and Expansible Multisensor Data Fusion Model for Perception in Level 3 Autonomous Driving System

Various sensors are adopted by autonomous driving systems to perceive objects and surroundings. Thus, the multi-sensor data fusion techniques become essential to combine different sensors’ advantages for better perception performance. However, the current multi-sensor data fusion techniques suffer from the high cost of computation resources, low expansibility for more diverse sensors, and insufficient systematic consideration for modeling. This paper first constructs a detachable and expansible multi-sensor data fusion model based on three main modules: front fusion, global fusion, and synthesizer, where the methods for flexible association gating and virtual targets have been designed. The model can be disassembled and configured for different trim levels of vehicles and is easily expansible for adding more heterogeneous sensors. Next, the presented multi-sensor data fusion model is compared with the cheap Joint Probabilistic Data Association (C-JPDA) method. The comparison shows the superior accuracy of the designed model on false association and effective narrowing of the variance of object detection. Finally, the presented multi-sensor data fusion model is integrated into an embedded system and experimented on urban roads and highways with the engaged Level 3 autonomous driving function. The experiment results indicate that the proposed model has excellent sensor data fusion performance and provides accurate and timely object information in the Level 3 autonomous driving system.

[1]  Mengsun Yu,et al.  Brain-Computer Interface Using Brain Power Map and Cognition Detection Network During Flight , 2022, IEEE/ASME Transactions on Mechatronics.

[2]  Qian Meng,et al.  A Multimodality Fusion Deep Neural Network and Safety Test Strategy for Intelligent Vehicles , 2021, IEEE Transactions on Intelligent Vehicles.

[3]  Fei Liu,et al.  Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[4]  Alberto Ferreira de Souza,et al.  Self-Driving Cars: A Survey , 2019, Expert Syst. Appl..

[5]  Huayan Pu,et al.  Comparative Study of Trajectory Tracking Control for Automated Vehicles via Model Predictive Control and Robust H-infinity State Feedback Control , 2020, Chinese Journal of Mechanical Engineering.

[6]  Hesheng Wang,et al.  Deep Learning-Based Localization and Perception Systems: Approaches for Autonomous Cargo Transportation Vehicles in Large-Scale, Semiclosed Environments , 2020, IEEE Robotics & Automation Magazine.

[7]  Larry S. Davis,et al.  DOPS: Learning to Detect 3D Objects and Predict Their 3D Shapes , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Ryo Yanase,et al.  Automated driving recognition technologies for adverse weather conditions , 2019 .

[9]  Tae-Hyoung Park,et al.  Fusing Lidar, Radar, and Camera Using Extended Kalman Filter for Estimating the Forward Position of Vehicles , 2019, 2019 IEEE International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE Conference on Robotics, Automation and Mechatronics (RAM).

[10]  Rahul Kumar,et al.  Radar and Camera Sensor Fusion with ROS for Autonomous Driving , 2019, 2019 Fifth International Conference on Image Information Processing (ICIIP).

[11]  Markus Lienkamp,et al.  A Deep Learning-based Radar and Camera Sensor Fusion Architecture for Object Detection , 2019, 2019 Sensor Data Fusion: Trends, Solutions, Applications (SDF).

[12]  Siyang Cao,et al.  Extending Reliability of mmWave Radar Tracking and Detection via Fusion With Camera , 2019, IEEE Access.

[13]  Ismail Hamieh,et al.  Construction of Autonomous Driving Maps employing LiDAR Odometry , 2019, 2019 IEEE Canadian Conference of Electrical and Computer Engineering (CCECE).

[14]  I. Bilik,et al.  Mitigation of Vehicle Vibration Effect on Automotive Radar , 2019, 2019 IEEE Radar Conference (RadarConf).

[15]  Wei-Yun Yau,et al.  Vehicle Detection and Width Estimation in Rain by Fusing Radar and Vision , 2018, 2018 15th International Conference on Control, Automation, Robotics and Vision (ICARCV).

[16]  Kwangsuck Boo,et al.  A Study on Real-Time Detection Method of Lane and Vehicle for Lane Change Assistant System Using Vision System on Highway , 2018, Engineering Science and Technology, an International Journal.

[17]  Bin Dai,et al.  Radar-Vision Fusion for Correcting the Position of Target Vehicles , 2018, 2018 10th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC).

[18]  Dong Seog Han,et al.  Radar and Vision Sensor Fusion for Object Detection in Autonomous Vehicle Surroundings , 2018, 2018 Tenth International Conference on Ubiquitous and Future Networks (ICUFN).

[19]  Jianqiang Wang,et al.  Object Classification Using CNN-Based Fusion of Vision and LIDAR in Autonomous Vehicle Environment , 2018, IEEE Transactions on Industrial Informatics.

[20]  Fabio Oleari,et al.  The PAN-Robots Project: Advanced Automated Guided Vehicle Systems for Industrial Logistics , 2018, IEEE Robotics & Automation Magazine.

[21]  Yi Yang,et al.  Real-Time Obstacles Detection and Status Classification for Collision Warning in a Vehicle Active Safety System , 2018, IEEE Transactions on Intelligent Transportation Systems.

[22]  Hui Xiong,et al.  Detection-level fusion for multi-object perception in dense traffic environment , 2017, 2017 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI).

[23]  Myo-Taeg Lim,et al.  Sensor fusion for vehicle tracking with camera and radar sensor , 2017, 2017 17th International Conference on Control, Automation and Systems (ICCAS).

[24]  Ivan Markovic,et al.  Radar and stereo vision fusion for multitarget tracking on the special Euclidean group , 2016, Robotics Auton. Syst..

[25]  Bin Yang,et al.  Fusion of stereo camera and MIMO-FMCW radar for pedestrian tracking in indoor environments , 2016, 2016 19th International Conference on Information Fusion (FUSION).

[26]  Jin Hyung Kim,et al.  Experimental studies of autonomous driving of a vehicle on the road using LiDAR and DGPS , 2015, 2015 15th International Conference on Control, Automation and Systems (ICCAS).

[27]  Torbjörn Ekman,et al.  Tracking With a High-Resolution 2D Spectral Estimation Based Automotive Radar , 2015, IEEE Transactions on Intelligent Transportation Systems.

[28]  Moongu Jeon,et al.  Data fusion of radar and image measurements for multi-object tracking via Kalman filtering , 2014, Inf. Sci..

[29]  Giorgio Battistelli,et al.  Distributed peer-to-peer multitarget tracking with association-based track fusion , 2014, 17th International Conference on Information Fusion (FUSION).

[30]  Akihiro Takeuchi,et al.  On-Road Multivehicle Tracking Using Deformable Object Model and Particle Filter With Improved Likelihood Estimation , 2012, IEEE Transactions on Intelligent Transportation Systems.

[31]  Wei Zhang,et al.  Near Neighbor Cheap JPDA IMM based on amplitude information , 2012, The 2012 International Workshop on Microwave and Millimeter Wave Circuits and System Technology.

[32]  Simon Winberg,et al.  Multisensor data fusion: Target tracking with a doppler radar and an Electro-Optic camera , 2011, 2011 IEEE International Conference on Control System, Computing and Engineering.

[33]  E. Richter,et al.  Radar and vision based data fusion - Advanced filtering techniques for a multi object vehicle tracking system , 2008, 2008 IEEE Intelligent Vehicles Symposium.

[34]  Feng Liu,et al.  IMMPDA vehicle tracking system using asynchronous sensor fusion of radar and vision , 2008, 2008 IEEE Intelligent Vehicles Symposium.

[35]  Alberto Broggi,et al.  Vehicle and Guard Rail Detection Using Radar and Vision Data Fusion , 2007, IEEE Transactions on Intelligent Transportation Systems.

[36]  Christian Hoffmann,et al.  Cheap Joint Probabilistic Data Association Filters in an Interacting Multiple Model Design , 2006, 2006 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems.

[37]  Zhijian Hu,et al.  Evaluation of data association techniques in a real multitarget radar tracking environment , 1995, Optics & Photonics.

[38]  Yaakov Bar-Shalom,et al.  Adaptive nonlinear filtering for tracking with measurements of uncertain origin , 1972, CDC 1972.

[39]  Allen J. Kanyuck,et al.  Correlation of Multiple-Site Track Data , 1970, IEEE Transactions on Aerospace and Electronic Systems.

[40]  Robert W. Sittler,et al.  An Optimal Data Association Problem in Surveillance Theory , 1964, IEEE Transactions on Military Electronics.

[41]  Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles , 2022 .