A Survey on Theories and Applications for Self-Driving Cars Based on Deep Learning Methods

Self-driving cars are a hot research topic in science and technology, which has a great influence on social and economic development. Deep learning is one of the current key areas in the field of artificial intelligence research. It has been widely applied in image processing, natural language understanding, and so on. In recent years, more and more deep learning-based solutions have been presented in the field of self-driving cars and have achieved outstanding results. This paper presents a review of recent research on theories and applications of deep learning for self-driving cars. This survey provides a detailed explanation of the developments of self-driving cars and summarizes the applications of deep learning methods in the field of self-driving cars. Then the main problems in self-driving cars and their solutions based on deep learning methods are analyzed, such as obstacle detection, scene recognition, lane detection, navigation and path planning. In addition, the details of some representative approaches for self-driving cars using deep learning methods are summarized. Finally, the future challenges in the applications of deep learning for self-driving cars are given out.

[1]  Alberto Testolin,et al.  Letter perception emerges from unsupervised deep learning and recycling of natural image features , 2017, Nature Human Behaviour.

[2]  Nakwan Kim,et al.  Collision avoidance for an unmanned surface vehicle using deep reinforcement learning , 2020 .

[3]  Shiliang Sun,et al.  A review of natural language processing techniques for opinion mining systems , 2017, Inf. Fusion.

[4]  Lawrie Jones.  DRIVERLESS CARS: WHEN AND WHERE , 2017 .

[5]  Jun Li,et al.  Radar and vision fusion for the real-time obstacle detection and identification , 2019, Ind. Robot.

[6]  Sriram Ganapathy,et al.  Modulation Filter Learning Using Deep Variational Networks for Robust Speech Recognition , 2019, IEEE Journal of Selected Topics in Signal Processing.

[7]  Fei Wu,et al.  Multi-view semantic learning network for point cloud based 3D object detection , 2020, Neurocomputing.

[8]  Maurizio Sangermano,et al.  A Study on Cyclists behaviour and bicycles Kinematic , 2020 .

[9]  Shuicheng Yan,et al.  A survey on deep learning-based fine-grained object classification and semantic segmentation , 2017, International Journal of Automation and Computing.

[10]  Sudha Natarajan,et al.  DeepRange: deep‐learning‐based object detection and ranging in autonomous driving , 2019, IET Intelligent Transport Systems.

[11]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[12]  Ju H. Park,et al.  Traffic Light Recognition Based on Binary Semantic Segmentation Network , 2019, Sensors.

[13]  Zhiming Luo,et al.  Spectral–Spatial Residual Network for Hyperspectral Image Classification: A 3-D Deep Learning Framework , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[14]  Dimitris Kiritsis,et al.  Deep learning for big data applications in CAD and PLM - Research review, opportunities and case study , 2018, Comput. Ind..

[15]  Fouzi Harrou,et al.  Obstacle Detection for Intelligent Transportation Systems Using Deep Stacked Autoencoder and $k$ -Nearest Neighbor Scheme , 2018, IEEE Sensors Journal.

[16]  Li Wang,et al.  Fusion of images and point clouds for the semantic segmentation of large-scale 3D scenes based on deep learning , 2018, ISPRS Journal of Photogrammetry and Remote Sensing.

[17]  Jingyi Liu,et al.  Learning full-reference quality-guided discriminative gradient cues for lane detection based on neural networks , 2019, J. Vis. Commun. Image Represent..

[18]  Luca Mantecchini,et al.  Analysis of the Relationship between Turning Signal Detection and Motorcycle Driver’s Characteristics on Urban Roads; A Case Study , 2019, Sensors.

[19]  Björn W. Schuller,et al.  Deep Learning for Environmentally Robust Speech Recognition , 2017, ACM Trans. Intell. Syst. Technol..

[20]  Toby P. Breckon,et al.  Quantitative Evaluation of Stereo Visual Odometry for Autonomous Vessel Localisation in Inland Waterway Sensing Applications , 2015, Sensors.

[21]  Jixin Liu,et al.  Fusing Object Semantics and Deep Appearance Features for Scene Recognition , 2019, IEEE Transactions on Circuits and Systems for Video Technology.

[22]  Jiwen Lu,et al.  Scene recognition with objectness , 2018, Pattern Recognit..

[23]  Ling Wang,et al.  Assessment of the safety benefits of vehicles' advanced driver assistance, connectivity and low level automation systems. , 2018, Accident; analysis and prevention.

[24]  Ying Zhu,et al.  Various Frameworks and Libraries of Machine Learning and Deep Learning: A Survey , 2019, Archives of Computational Methods in Engineering.

[25]  Soon Ki Jung,et al.  Deep Neural Network Concepts for Background Subtraction: A Systematic Review and Comparative Evaluation , 2018, Neural Networks.

[26]  Limin Wang,et al.  Knowledge Guided Disambiguation for Large-Scale Scene Classification With Multi-Resolution CNNs , 2016, IEEE Transactions on Image Processing.

[27]  Zheng Liu,et al.  Real-time road surface and semantic lane estimation using deep features , 2018, Signal Image Video Process..

[28]  Emilio Frazzoli,et al.  A Survey of Motion Planning and Control Techniques for Self-Driving Urban Vehicles , 2016, IEEE Transactions on Intelligent Vehicles.

[29]  Ming Zhu,et al.  Obstacle detection in single images with deep neural networks , 2016, Signal Image Video Process..

[30]  Jae Wook Jeon,et al.  Wide context learning network for stereo matching , 2019, Signal Process. Image Commun..

[31]  Sherali Zeadally,et al.  Autonomous Cars: Research Results, Issues, and Future Challenges , 2019, IEEE Communications Surveys & Tutorials.

[32]  Gautam Srivastava,et al.  Automatic recognition algorithm of traffic signs based on convolution neural network , 2020, Multimedia Tools and Applications.

[33]  Jian Ma,et al.  Intelligent fault diagnosis for rotating machinery using deep Q-network based health state classification: A deep reinforcement learning approach , 2019, Adv. Eng. Informatics.

[34]  Xun Wang,et al.  Lane marking detection via deep convolutional neural network , 2017, Neurocomputing.

[35]  Alexander G. Mirnig,et al.  A taxonomy of autonomous vehicle handover situations , 2019 .

[36]  Yong Yu,et al.  A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures , 2019, Neural Computation.

[37]  Pascual Campoy Cervera,et al.  A Review of Deep Learning Methods and Applications for Unmanned Aerial Vehicles , 2017, J. Sensors.

[38]  Hsiang-Chieh Chen.  Monocular Vision-Based Obstacle Detection and Avoidance for a Multicopter , 2019, IEEE Access.

[39]  Pau-Choo Chung,et al.  Improved Lane Detection With Multilevel Features in Branch Convolutional Neural Networks , 2019, IEEE Access.

[40]  Pengfei Shi,et al.  A Dynamic Bioinspired Neural Network Based Real-Time Path Planning Method for Autonomous Underwater Vehicles , 2017, Comput. Intell. Neurosci..

[41]  Sparsh Mittal,et al.  A Survey on optimized implementation of deep learning models on the NVIDIA Jetson platform , 2019, J. Syst. Archit..

[42]  Guohui Tian,et al.  A novel scene classification model combining ResNet based transfer learning and data augmentation with a filter , 2019, Neurocomputing.

[43]  Michelle Birdsall.  Google and ITE: The Road Ahead for Self-Driving Cars , 2014 .

[44]  Jianru Xue,et al.  A Survey of Scene Understanding by Event Reasoning in Autonomous Driving , 2018, Int. J. Autom. Comput..

[45]  Guodong Guo,et al.  A survey on deep learning based face recognition , 2019, Comput. Vis. Image Underst..

[46]  Long Chen,et al.  Robust Lane Detection From Continuous Driving Scenes Using Deep Neural Networks , 2019, IEEE Transactions on Vehicular Technology.

[47]  Mohsen Guizani,et al.  Unmanned Aerial Vehicles (UAVs): A Survey on Civil Applications and Key Research Challenges , 2018, IEEE Access.

[48]  Long Chen,et al.  Deep Integration: A Multi-Label Architecture for Road Scene Recognition , 2019, IEEE Transactions on Image Processing.

[49]  Neeraj Kumar,et al.  Convolutional neural networks for 5G-enabled Intelligent Transportation System : A systematic review , 2020, Comput. Commun..

[50]  Jaafar Alghazo,et al.  Autonomous Traffic Sign (ATSR) Detection and Recognition using Deep CNN , 2019, Procedia Computer Science.

[51]  Cheng Soon Ong,et al.  Multivariate spearman's ρ for aggregating ranks using copulas , 2016 .

[52]  Hye-Jin Kim,et al.  Exploring the key services and players in the smart car market , 2019, Telecommunications Policy.

[53]  Paolo Valigi,et al.  J-MOD2: Joint Monocular Obstacle Detection and Depth Estimation , 2017, IEEE Robotics and Automation Letters.

[54]  Fabio Duarte,et al.  Self-driving cars: A city perspective , 2019, Science Robotics.

[55]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[56]  Zubair Khan,et al.  A Memetic Algorithm with variable Length chromosome for robot Path Planning under Dynamic Environments , 2017, Int. J. Robotics Autom..

[57]  Leonard Barolli,et al.  Fuzzy-based Driver Monitoring System (FDMS): Implementation of two intelligent FDMSs and a testbed for safe driving in VANETs , 2020, Future Gener. Comput. Syst..

[58]  Dacheng Tao,et al.  Deep Neural Network for Structural Prediction and Lane Detection in Traffic Scene , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[59]  Hai-Jun Su,et al.  A feedforward and feedback integrated lateral and longitudinal driver model for personalized advanced driver assistance systems , 2018 .

[60]  Fuchun Sun,et al.  Robotic Material Perception Using Active Multimodal Fusion , 2019, IEEE Transactions on Industrial Electronics.

[61]  Tiago M. Fernández-Caramés,et al.  A Review on IoT Deep Learning UAV Systems for Autonomous Obstacle Detection and Collision Avoidance , 2019, Remote. Sens..

[62]  Mingliang Liu,et al.  Accelerating geostatistical seismic inversion using TensorFlow: A heterogeneous distributed deep learning framework , 2019, Comput. Geosci..

[63]  Yann LeCun,et al.  Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches , 2015, J. Mach. Learn. Res..

[64]  Nathan A. Greenblatt,et al.  Self-driving cars and the law , 2016, IEEE Spectrum.

[65]  Simon X. Yang,et al.  Bioinspired Intelligent Algorithm and Its Applications for Mobile Robot Control: A Survey , 2015, Comput. Intell. Neurosci..

[66]  Kara M. Kockelman,et al.  Economic Effects of Automated Vehicles , 2017 .

[67]  Daijin Kim,et al.  Accurate traffic light detection using deep neural network with focal regression loss , 2019, Image Vis. Comput..

[68]  Erik Cambria,et al.  Recent Trends in Deep Learning Based Natural Language Processing , 2017, IEEE Comput. Intell. Mag..

[69]  Zhiwei Cao,et al.  Multi-Level Ensemble Network for Scene Recognition , 2019, Multimedia Tools and Applications.

[70]  Aditya Khamparia,et al.  A systematic review on deep learning architectures and applications , 2019, Expert Syst. J. Knowl. Eng..

[71]  Hanqing Lu,et al.  Contextual deconvolution network for semantic segmentation , 2020, Pattern Recognit..

[72]  Simon X. Yang,et al.  Bioinspired Neural Network-based Q-Learning Approach for robot Path Planning in unknown Environments , 2016, Int. J. Robotics Autom..

[73]  Chao Shen,et al.  Multi‐receptive field graph convolutional neural networks for pedestrian detection , 2019, IET Intelligent Transport Systems.

[74]  Jiamao Li,et al.  Multi-Dimensional Residual Dense Attention Network for Stereo Matching , 2019, IEEE Access.

[75]  Li Ren,et al.  A Multiagent Q-Learning-Based Optimal Allocation Approach for Urban Water Resource Management System , 2014, IEEE Transactions on Automation Science and Engineering.

[76]  Dan Zhang,et al.  Path planning for active SLAM based on deep reinforcement learning under unknown environments , 2020, Intell. Serv. Robotics.

[77]  Liu Yang,et al.  An Improved Spinal Neural System-Based Approach for Heterogeneous AUVs Cooperative Hunting , 2018, Int. J. Fuzzy Syst..

[78]  Sam Kwong,et al.  G-MS2F: GoogLeNet based multi-stage feature fusion of deep CNN for scene recognition , 2017, Neurocomputing.

[79]  Degui Xiao,et al.  Attention deep neural network for lane marking detection , 2020, Knowl. Based Syst..

[80]  Jonas Nilsson,et al.  Driving tests for self-driving cars , 2018, IEEE Spectrum.

[81]  Lingli Yu,et al.  Intelligent Land-Vehicle Model Transfer Trajectory Planning Method Based on Deep Reinforcement Learning , 2018, Sensors.

[82]  Minho Lee,et al.  Fast learning method for convolutional neural networks using extreme learning machine and its application to lane detection , 2017, Neural Networks.