Multi-Modal Fusion Technology Based on Vehicle Information: A Survey

Multi-modal fusion is a basic task of autonomous driving system perception, which has attracted many scholars' interest in recent years. The current multi-modal fusion methods mainly focus on camera data and LiDAR data, but pay little attention to the kinematic information provided by the bottom sensors of the vehicle, such as acceleration, vehicle speed, angle of rotation. These information are not affected by complex external scenes, so it is more robust and reliable. In this paper, we introduce the existing application fields of vehicle bottom information and the research progress of related methods, as well as the multi-modal fusion methods based on bottom information. We also introduced the relevant information of the vehicle bottom information data set in detail to facilitate the research as soon as possible. In addition, new future ideas of multi-modal fusion technology for autonomous driving tasks are proposed to promote the further utilization of vehicle bottom information.

[1]  Santanu Chaudhury,et al.  DriveBFR: Driver Behavior and Fuel-Efficiency-Based Recommendation System , 2022, IEEE Transactions on Computational Social Systems.

[2]  X. Zhang,et al.  OpenMPD: An Open Multimodal Perception Dataset for Autonomous Driving , 2022, IEEE Transactions on Vehicular Technology.

[3]  Shenmin Zhang,et al.  Prediction of Vehicle Braking Deceleration Based on BP Neural Network , 2022, Journal of Physics: Conference Series.

[4]  A. Karar,et al.  Machine learning methods for driver behaviour classification , 2021, 2021 4th International Conference on Bio-Engineering for Smart Technologies (BioSMART).

[5]  Jiaming Xing,et al.  Energy Management Strategy Based on a Novel Speed Prediction Method , 2021, Sensors.

[6]  I. Boulkaibet,et al.  Driver Behavior Classification System Analysis Using Machine Learning Methods , 2021, Applied Sciences.

[7]  Jiaming Xing,et al.  Dual-Input and Multi-Channel Convolutional Neural Network Model for Vehicle Speed Prediction , 2021, Sensors.

[8]  Haluk Kucuk,et al.  Driver Profiling Using Long Short Term Memory (LSTM) and Convolutional Neural Network (CNN) Methods , 2021, IEEE Transactions on Intelligent Transportation Systems.

[9]  Chen Lv,et al.  Improved Short-Term Speed Prediction Using Spatiotemporal-Vision-Based Deep Neural Network for Intelligent Fuel Cell Vehicles , 2021, IEEE Transactions on Industrial Informatics.

[10]  Peer Neubert,et al.  Multivariate Time Series Analysis for Driving Style Classification using Neural Networks and Hyperdimensional Computing , 2021, 2021 IEEE Intelligent Vehicles Symposium (IV).

[11]  Siyu Zhu,et al.  CondLaneNet: a Top-to-down Lane Detection Framework Based on Conditional Convolution , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[12]  Chen Lv,et al.  Real-Time Optimization of Energy Management Strategy for Fuel Cell Vehicles Using Inflated 3D Inception Long Short-Term Memory Network-Based Speed Prediction , 2021, IEEE Transactions on Vehicular Technology.

[13]  Thiago Oliveira-Santos,et al.  Keep your Eyes on the Lane: Real-time Attention-guided Lane Detection , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Deng Cai,et al.  RESA: Recurrent Feature-Shift Aggregator for Lane Detection , 2020, AAAI.

[15]  Jingda Wu,et al.  Multi-Modal Sensor Fusion-Based Deep Neural Network for End-to-End Autonomous Driving With Scene Understanding , 2020, IEEE Sensors Journal.

[16]  Gaurav Sahu,et al.  Adaptive Fusion Techniques for Multimodal Data , 2021, EACL.

[17]  Yu Shi,et al.  Multimodal Fusion Method Based on Self-Attention Mechanism , 2020, Wirel. Commun. Mob. Comput..

[18]  Arno Eichberger,et al.  Applying deep neural networks for multi-level classification of driver drowsiness using Vehicle-based measures , 2020, Expert Syst. Appl..

[19]  Mahmood Fathy,et al.  Driver behavior detection and classification using deep convolutional neural networks , 2020, Expert Syst. Appl..

[20]  Lisardo Prieto González,et al.  Simultaneous Estimation of Vehicle Roll and Sideslip Angles through a Deep Learning Approach , 2020, Sensors.

[21]  Huanyu Wang,et al.  Ultra Fast Structure-aware Deep Lane Detection , 2020, ECCV.

[22]  Yohannes Kassahun,et al.  A2D2: Audi Autonomous Driving Dataset , 2020, ArXiv.

[23]  Dragomir Anguelov,et al.  Scalability in Perception for Autonomous Driving: Waymo Open Dataset , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Qiang Xu,et al.  nuScenes: A Multimodal Dataset for Autonomous Driving , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Felix Heide,et al.  Seeing Through Fog Without Seeing Fog: Deep Multimodal Sensor Fusion in Unseen Adverse Weather , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Trevor Darrell,et al.  BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning , 2018, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Ruigang Yang,et al.  The ApolloScape Open Dataset for Autonomous Driving and Its Application , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Prasenjit Basak,et al.  Estimation of energy consumption of electric vehicles using Deep Convolutional Neural Network to reduce driver's range anxiety. , 2020, ISA transactions.

[29]  Datong Qin,et al.  Driving Intention Identification Based on Long Short-Term Memory and A Case Study in Shifting Strategy Optimization , 2019, IEEE Access.

[30]  Yong Wang,et al.  Short-term Vehicle Speed Prediction by Time Series Neural Network in High Altitude Areas , 2019, IOP Conference Series: Earth and Environmental Science.

[31]  Myoungho Sunwoo,et al.  Ego-Vehicle Speed Prediction Using a Long Short-Term Memory Based Recurrent Neural Network , 2019, International Journal of Automotive Technology.

[32]  Wen-Hua Chen,et al.  A machine learning based personalized system for driving state recognition , 2019, Transportation Research Part C: Emerging Technologies.

[33]  Xiangmo Zhao,et al.  Long short‐term memory and convolutional neural network for abnormal driving behaviour recognition , 2019, IET Intelligent Transport Systems.

[34]  Dan Wang,et al.  End-to-End Self-Driving Using Deep Neural Networks with Multi-auxiliary Tasks , 2019, Automotive Innovation.

[35]  Simon Lucey,et al.  Argoverse: 3D Tracking and Forecasting With Rich Maps , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Shobit Sharma,et al.  Lateral and Longitudinal Motion Control of Autonomous Vehicles using Deep Learning , 2019, 2019 IEEE International Conference on Electro Information Technology (EIT).

[37]  Michael E. Fitzpatrick,et al.  Instantaneous vehicle fuel consumption estimation using smartphones and recurrent neural networks , 2019, Expert Syst. Appl..

[38]  Xindong Wu,et al.  Object Detection With Deep Learning: A Review , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[39]  Jianhai Zhang,et al.  Deep Multimodal Multilinear Fusion with High-order Polynomial Pooling , 2019, NeurIPS.

[40]  Shobit Sharma,et al.  Behavioral Cloning for Lateral Motion Control of Autonomous Vehicles Using Deep Learning , 2018, 2018 IEEE International Conference on Electro/Information Technology (EIT).

[41]  Louis-Philippe Morency,et al.  Efficient Low-rank Multimodal Fusion With Modality-Specific Factors , 2018, ACL.

[42]  Lei Gao,et al.  Discriminative Multiple Canonical Correlation Analysis for Information Fusion , 2018, IEEE Transactions on Image Processing.

[43]  Luc Van Gool,et al.  Towards End-to-End Lane Detection: an Instance Segmentation Approach , 2018, 2018 IEEE Intelligent Vehicles Symposium (IV).

[44]  Jiebo Luo,et al.  End-to-end Multi-Modal Multi-Task Vehicle Control for Self-Driving Cars with Visual Perceptions , 2018, 2018 24th International Conference on Pattern Recognition (ICPR).

[45]  Andrew Zisserman,et al.  Objects that Sound , 2017, ECCV.

[46]  Xiaogang Wang,et al.  Spatial As Deep: Spatial CNN for Traffic Scene Understanding , 2017, AAAI.

[47]  Paulo Peixoto,et al.  Multimodal vehicle detection: fusing 3D-LIDAR and color camera data , 2017, Pattern Recognit. Lett..

[48]  Wei Zhan,et al.  Fusing Bird View LIDAR Point Cloud and Front View Camera Image for Deep Object Detection , 2017, ArXiv.

[49]  Hesham M. Eraqi,et al.  End-to-End Deep Learning for Steering Autonomous Vehicles Considering Temporal Dependencies , 2017, ArXiv.

[50]  Saeid Nahavandi,et al.  Driving behavior classification based on sensor data fusion using LSTM recurrent neural networks , 2017, 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC).

[51]  Sang-Sun Lee,et al.  Long-term prediction of vehicle trajectory based on a deep neural network , 2017, 2017 International Conference on Information and Communication Technology Convergence (ICTC).

[52]  Erik Cambria,et al.  Tensor Fusion Network for Multimodal Sentiment Analysis , 2017, EMNLP.

[53]  Yunsi Fei,et al.  Vehicle Speed Prediction by Two-Level Data Driven Models in Vehicular Networks , 2017, IEEE Transactions on Intelligent Transportation Systems.

[54]  Chang Liu,et al.  Learning a deep neural net policy for end-to-end control of autonomous vehicles , 2017, 2017 American Control Conference (ACC).

[55]  Andrew Zisserman,et al.  Look, Listen and Learn , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[56]  Lonce L. Wyse,et al.  Audio Spectrogram Representations for Processing with Convolutional Neural Networks , 2017, ArXiv.

[57]  José García Rodríguez,et al.  A Review on Deep Learning Techniques Applied to Semantic Segmentation , 2017, ArXiv.

[58]  Paul Newman,et al.  1 year, 1000 km: The Oxford RobotCar dataset , 2017, Int. J. Robotics Res..

[59]  Hang-Bong Kang,et al.  Object Detection and Classification by Decision-Level Fusion for Intelligent Vehicle Systems , 2017, Sensors.

[60]  Ji Wan,et al.  Multi-view 3D Object Detection Network for Autonomous Driving , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[61]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[62]  Jürgen Schmidhuber,et al.  LSTM: A Search Space Odyssey , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[63]  Luis Miguel Bergasa,et al.  Need data for driver behaviour analysis? Presenting the public UAH-DriveSet , 2016, 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC).

[64]  Robert Weidner,et al.  Telematic driving profile classification in car insurance pricing , 2016, Annals of Actuarial Science.

[65]  Eder Santana,et al.  Learning a Driving Simulator , 2016, ArXiv.

[66]  Antonio M. López,et al.  The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[67]  Zsolt Kira,et al.  Fusing LIDAR and images for pedestrian detection using convolutional neural networks , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[68]  Xin Zhang,et al.  End to End Learning for Self-Driving Cars , 2016, ArXiv.

[69]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[70]  Gys Albertus Marthinus Meiring,et al.  A Review of Intelligent Driving Style Analysis Systems and Related Artificial Intelligence Algorithms , 2015, Sensors.

[71]  Patrick Bouthemy,et al.  Optical flow modeling and computation: A survey , 2015, Comput. Vis. Image Underst..

[72]  Karl Heinz Hoffmann,et al.  A mathematical model for predicting lane changes using the steering wheel angle. , 2014, Journal of safety research.

[73]  Andreas Geiger,et al.  Vision meets robotics: The KITTI dataset , 2013, Int. J. Robotics Res..

[74]  John H. L. Hansen,et al.  Leveraging sensor information from portable devices towards automatic driving maneuver recognition , 2012, 2012 15th International IEEE Conference on Intelligent Transportation Systems.

[75]  Mateu Sbert,et al.  Multimodal Data Fusion Based on Mutual Information , 2012, IEEE Transactions on Visualization and Computer Graphics.

[76]  Chih-Hsien Huang,et al.  Artificial neural network for predictions of vehicle drivable range and period , 2012, 2012 IEEE International Conference on Vehicular Electronics and Safety (ICVES 2012).

[77]  Joonwoo Son,et al.  Relationships between Driving Style and Fuel Consumption in Highway Driving , 2011 .

[78]  Edoardo Sabbioni,et al.  On the vehicle sideslip angle estimation through neural networks: Numerical and experimental results , 2011 .

[79]  Shekhar Verma,et al.  Prediction of Lane Change Trajectories through Neural Network , 2010, 2010 International Conference on Computational Intelligence and Communication Networks.

[80]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[81]  A. Vicino,et al.  Nonlinear time series analysis of dissolved oxygen in the Orbetello Lagoon (Italy) , 2007 .

[82]  Joseph P. Zbilut,et al.  Recurrence quantification analysis and state space divergence reconstruction for financial time series analysis , 2007 .

[83]  Jorge Belaire-Franch,et al.  Assessing nonlinear structures in real exchange rates using recurrence plot strategies , 2002 .

[84]  Azim Eskandarian,et al.  Unobtrusive drowsiness detection by neural network learning of driver steering , 2001 .