Artificial intelligence applications in the development of autonomous vehicles: a survey

The advancement of artificial intelligence ( AI ) has truly stimulated the development and deployment of autonomous vehicles ( AVs ) in the transportation industry. Fueled by big data from various sensing devices and advanced computing resources, AI has become an essential component of AVs for perceiving the surrounding environment and making appropriate decision in motion. To achieve goal of full automation ( i.e., self-driving ( , it is important to know how AI works in AV systems. Existing research have made great efforts in investigating different aspects of applying AI in AV development. However, few studies have offered the research community a thorough examination of current practices in implementing AI in AVs. Thus, this paper aims to shorten the gap by providing a comprehensive survey of key studies in this research avenue. Specifically, it intends to analyze their use of AIs in supporting the primary applications in AVs: 1) perception; 2) localization and mapping; and 3) decision making. It investigates the current practices to understand how AI can be used and what are the challenges and issues associated with their implementation. Based on the exploration of current practices and technology advances, this paper further provides insights into potential opportunities regarding the use of AI in conjunction with other emerging technologies: 1) high definition maps, big data, and high performance computing; 2) augmented reality( AR ) / virtual reality ( VR ) enhanced simulation platform; and 3) 5G communication for connected AVs. This paper is expected to offer a quick reference for researchers interested in understanding the use of AI in AV research.

[1]  Mark Campbell,et al.  All Weather Perception: Joint Data Association, Tracking, and Classification for Autonomous Ground Vehicles , 2016, ArXiv.

[2]  Xinming Huang,et al.  Accurate and Reliable Detection of Traffic Lights Using Multiclass Learning and Multiobject Tracking , 2016, IEEE Intelligent Transportation Systems Magazine.

[3]  Karin Strauss,et al.  Accelerating Deep Convolutional Neural Networks Using Specialized Hardware , 2015 .

[4]  Jason Cong,et al.  Energy-Efficient CNN Implementation on a Deeply Pipelined FPGA Cluster , 2016, ISLPED.

[5]  Lawrence D. Jackel,et al.  Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car , 2017, ArXiv.

[6]  Hong Wang,et al.  Parallel planning: a new motion planning framework for autonomous driving , 2019, IEEE/CAA Journal of Automatica Sinica.

[7]  John M. Dolan,et al.  Automatically Generated Curriculum based Reinforcement Learning for Autonomous Vehicles in Urban Environment , 2018, 2018 IEEE Intelligent Vehicles Symposium (IV).

[8]  Diego Alonso,et al.  A Machine Learning Approach to Pedestrian Detection for Autonomous Vehicles Using High-Definition 3D Range Data , 2016, Sensors.

[9]  Ragunathan Rajkumar,et al.  Towards a viable autonomous driving research platform , 2013, 2013 IEEE Intelligent Vehicles Symposium (IV).

[10]  Dushyant Rao,et al.  Deep tracking in the wild: End-to-end tracking using recurrent neural networks , 2018, Int. J. Robotics Res..

[11]  Xin Li,et al.  Algorithm and hardware implementation for visual perception system in autonomous vehicle: A survey , 2017, Integr..

[12]  Vicente Milanés Montero,et al.  Automatic lateral control for unmanned vehicles via genetic algorithms , 2011, Appl. Soft Comput..

[13]  George J. Vachtsevanos,et al.  Intelligent route planning for fast autonomous vehicles operating in a large natural terrain , 2002, Robotics Auton. Syst..

[14]  Qiang Zhang,et al.  Driving decision-making analysis of car-following for autonomous vehicle under complex urban environment , 2016, 2016 9th International Symposium on Computational Intelligence and Design (ISCID).

[15]  Jiajia Chen,et al.  A path planning method of anti-jamming ability improvement for autonomous vehicle navigating in off-road environments , 2017, Ind. Robot.

[16]  Dean Pomerleau,et al.  ALVINN, an autonomous land vehicle in a neural network , 2015 .

[17]  Anja Klein,et al.  An Online Context-Aware Machine Learning Algorithm for 5G mmWave Vehicular Communications , 2018, IEEE/ACM Transactions on Networking.

[18]  Erik Frisk,et al.  Deep Learning Model Predictive Control for Autonomous Driving in Unknown Environments , 2018, SyRoCo.

[19]  Shinpei Kato,et al.  An Open Approach to Autonomous Vehicles , 2015, IEEE Micro.

[20]  Robert W. Heath,et al.  Millimeter-Wave Vehicular Communication to Support Massive Automotive Sensing , 2016, IEEE Communications Magazine.

[21]  Wei Li,et al.  Vision navigation of an autonomous vehicle by fuzzy reasoning , 1998, 9th European Signal Processing Conference (EUSIPCO 1998).

[22]  Fouzi Harrou,et al.  Unsupervised obstacle detection in driving environments using deep-learning-based stereovision , 2018, Robotics Auton. Syst..

[23]  Sebastian Thrun,et al.  Model based vehicle detection and tracking for autonomous urban driving , 2009, Auton. Robots.

[24]  Oleg Olegovich Varlamov,et al.  Experimental Autonomous Road Vehicle with Logical Artificial Intelligence , 2017 .

[25]  Qichao Zhang,et al.  Multi-task learning for dangerous object detection in autonomous driving , 2017, Inf. Sci..

[26]  Zhe Wu,et al.  Deep Learning Technique-Based Steering of Autonomous Car , 2018, Int. J. Comput. Intell. Appl..

[27]  Alois Knoll,et al.  Uncertainty in Machine Learning: A Safety Perspective on Autonomous Driving , 2018, SAFECOMP Workshops.

[28]  Bo Cheng,et al.  Car-following method based on inverse reinforcement learning for autonomous vehicle decision-making , 2018, International Journal of Advanced Robotic Systems.

[29]  Kazuhide Okamoto,et al.  Driver Intention-based Vehicle Threat Assessment using Random Forests and Particle Filtering , 2017 .

[30]  John McDonald,et al.  Computer vision in automated parking systems: Design, implementation and challenges , 2017, Image Vis. Comput..

[31]  Giulio Reina,et al.  Learning Traversability From Point Clouds in Challenging Scenarios , 2018, IEEE Transactions on Intelligent Transportation Systems.

[32]  Germán Ros,et al.  Street-view change detection with deconvolutional networks , 2016, Autonomous Robots.

[33]  Eric Sax,et al.  Machine learning and deep neural network — Artificial intelligence core for lab and real-world test and validation for ADAS and autonomous vehicles: AI for efficient and quality test and validation , 2017, 2017 Intelligent Systems Conference (IntelliSys).

[34]  Xin Zhang,et al.  End to End Learning for Self-Driving Cars , 2016, ArXiv.

[35]  Araz Taeihagh,et al.  Governing autonomous vehicles: emerging responses for safety, liability, privacy, cybersecurity, and industry risks , 2018, Transport Reviews.

[36]  Dayong Shen,et al.  Traffic Sign Recognition Using Kernel Extreme Learning Machines With Deep Perceptual Features , 2017, IEEE Transactions on Intelligent Transportation Systems.

[37]  Jian Yang,et al.  Importance-Aware Semantic Segmentation for Autonomous Vehicles , 2019, IEEE Transactions on Intelligent Transportation Systems.

[38]  Ryad Chellali,et al.  An Overview of Nature-Inspired, Conventional, and Hybrid Methods of Autonomous Vehicle Path Planning , 2018, Journal of Advanced Transportation.

[39]  Qingquan Li,et al.  Turn Signal Detection During Nighttime by CNN Detector and Perceptual Hashing Tracking , 2017, IEEE Transactions on Intelligent Transportation Systems.

[40]  Yuzhong Shen,et al.  AR-PED: A framework of augmented reality enabled pedestrian-in-the-loop simulation , 2019, Simul. Model. Pract. Theory.

[41]  Péter Gáspár,et al.  Highway Environment Model for Reinforcement Learning , 2018 .

[42]  Dominique Gruyer,et al.  Intelligent Vehicle Embedded Sensors Fault Detection and Isolation Using Analytical Redundancy and Nonlinear Transformations , 2017, J. Control. Sci. Eng..

[43]  Frank Vahid,et al.  Switching Predictive Control Using Reconfigurable State-Based Model , 2019, ACM Trans. Design Autom. Electr. Syst..

[44]  Zheng Liu,et al.  Saliency Map Generation by the Convolutional Neural Network for Real-Time Traffic Light Detection Using Template Matching , 2015, IEEE Transactions on Computational Imaging.

[45]  Christos Katrakazas,et al.  Real-time motion planning methods for autonomous on-road driving: State-of-the-art and future research directions , 2015 .

[46]  Chi-Kwong Li,et al.  An approach to tune fuzzy controllers based on reinforcement learning for autonomous vehicle control , 2005, IEEE Transactions on Intelligent Transportation Systems.

[47]  Jürgen Schmidhuber,et al.  Multi-column deep neural network for traffic sign classification , 2012, Neural Networks.

[48]  Michael Botsch,et al.  A Machine Learning Approach for the Segmentation of Driving Maneuvers and its Application in Autonomous Parking , 2017, J. Artif. Intell. Soft Comput. Res..

[49]  David A. Patterson,et al.  In-datacenter performance analysis of a tensor processing unit , 2017, 2017 ACM/IEEE 44th Annual International Symposium on Computer Architecture (ISCA).

[50]  Matthew Johnson-Roberson,et al.  Failing to Learn: Autonomously Identifying Perception Failures for Self-Driving Cars , 2017, IEEE Robotics and Automation Letters.

[51]  R. Maskeliūnas,et al.  A hybrid liar/radar-based deep learning and vehicle recognition engine for autonomous vehicle precrash control , 2018, Eastern-European Journal of Enterprise Technologies.

[52]  Soyoung Ahn,et al.  Platooning Strategy for Connected and Autonomous Vehicles: Transition from Light Traffic , 2017 .

[53]  Werner Huber,et al.  Experience, Results and Lessons Learned from Automated Driving on Germany's Highways , 2015, IEEE Intelligent Transportation Systems Magazine.

[54]  Shijie Li,et al.  Throughput-Optimized FPGA Accelerator for Deep Convolutional Neural Networks , 2017, ACM Trans. Reconfigurable Technol. Syst..

[55]  Hsin-Mu Tsai,et al.  A Vision-Based Hierarchical Framework for Autonomous Front-Vehicle Taillights Detection and Signal Recognition , 2015, 2015 IEEE 18th International Conference on Intelligent Transportation Systems.

[56]  Chang Liu,et al.  Scene Understanding in Deep Learning-Based End-to-End Controllers for Autonomous Vehicles , 2019, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[57]  Mianxiong Dong,et al.  Humanlike Driving: Empirical Decision-Making System for Autonomous Vehicles , 2018, IEEE Transactions on Vehicular Technology.

[58]  Amnon Shashua,et al.  Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving , 2016, ArXiv.

[59]  Péter Gáspár,et al.  Data-Driven Reachability Analysis for the Reconfiguration of Vehicle Control Systems , 2018 .

[60]  Daniel J. Fagnant,et al.  Preparing a Nation for Autonomous Vehicles: Opportunities, Barriers and Policy Recommendations , 2015 .

[61]  Jianxiong Xiao,et al.  DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[62]  Reza Langari,et al.  Autonomous Vehicles , 2016, Science.

[63]  Etienne Perot,et al.  Deep Reinforcement Learning framework for Autonomous Driving , 2017, Autonomous Vehicles and Machines.

[64]  Naohito Takasuka,et al.  Road Surface Recognition Using Laser Radar for Automatic Platooning , 2016, IEEE Transactions on Intelligent Transportation Systems.

[65]  Jin Zhang,et al.  Could We Issue Driving Licenses to Autonomous Vehicles? , 2018, SAFECOMP Workshops.

[66]  Yuzhong Shen,et al.  Framework for Interactive M3 Visualization of Microscopic Traffic Simulation , 2018, Transportation Research Record: Journal of the Transportation Research Board.

[67]  Leopoldo Acosta,et al.  A Combined Voxel and Particle Filter-Based Approach for Fast Obstacle Detection and Tracking in Automotive Applications , 2017, IEEE Transactions on Intelligent Transportation Systems.

[68]  Sudha Natarajan,et al.  Traffic sign recognition using weighted multi‐convolutional neural network , 2018, IET Intelligent Transport Systems.

[69]  Kyunghyun Cho,et al.  Query-Efficient Imitation Learning for End-to-End Autonomous Driving , 2016, ArXiv.

[70]  Klaus McDonald-Maier,et al.  Intelligent intrusion detection in external communication systems for autonomous vehicles , 2018 .

[71]  Alberto Sangiovanni-Vincentelli,et al.  Driving-Style-Based Codesign Optimization of an Automated Electric Vehicle: A Cyber-Physical System Approach , 2019, IEEE Transactions on Industrial Electronics.

[72]  Mehmed Kantardzic,et al.  Hybrid Automaton Based Vehicle Platoon Modelling and Cooperation Behaviour Profile Prediction , 2018 .

[73]  Lili Du,et al.  Cooperative platoon control for a mixed traffic flow including human drive vehicles and connected and autonomous vehicles , 2018, Transportation Research Part B: Methodological.

[74]  Xin Li,et al.  Efficient statistical validation of machine learning systems for autonomous driving , 2016, 2016 IEEE/ACM International Conference on Computer-Aided Design (ICCAD).

[75]  Fernando Santos Osório,et al.  Adaptive finite state machine based visual autonomous navigation system , 2014, Eng. Appl. Artif. Intell..

[76]  Yu-an Zhang,et al.  A Block Object Detection Method Based on Feature Fusion Networks for Autonomous Vehicles , 2019, Complex..

[77]  Ching-Yao Chan,et al.  Formulation of deep reinforcement learning architecture toward autonomous driving for on-ramp merge , 2017, 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC).

[78]  Jiujun Cheng,et al.  Deep learning–based traffic sign recognition for unmanned autonomous vehicles , 2018, J. Syst. Control. Eng..

[79]  Hesham M. Eraqi,et al.  End-to-End Deep Learning for Steering Autonomous Vehicles Considering Temporal Dependencies , 2017, ArXiv.

[80]  Jianping Fan,et al.  Re-ranking Object Proposals for Object Detection in Automatic Driving , 2016, ArXiv.

[81]  Jiaolong Xu,et al.  Training my car to see using virtual worlds , 2017, Image Vis. Comput..

[82]  Zhidong Deng,et al.  Segmentation of Drivable Road Using Deep Fully Convolutional Residual Network with Pyramid Pooling , 2018, Cognitive Computation.

[83]  Nanning Zheng,et al.  Springrobot: a prototype autonomous vehicle and its algorithms for lane detection , 2004, IEEE Transactions on Intelligent Transportation Systems.

[84]  Matthew Johnson-Roberson,et al.  Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks? , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[85]  Emilio Frazzoli,et al.  The Impact of Cooperative Perception on Decision Making and Planning of Autonomous Vehicles , 2015, IEEE Intelligent Transportation Systems Magazine.

[86]  Homayoun Najjaran,et al.  Autonomous vehicle perception: The technology of today and tomorrow , 2018 .

[87]  Wael Farag,et al.  Recognition of traffic signs by convolutional neural nets for self-driving vehicles , 2018, Int. J. Knowl. Based Intell. Eng. Syst..

[88]  YiNa Jeong,et al.  An Integrated Self-Diagnosis System for an Autonomous Vehicle Based on an IoT Gateway and Deep Learning , 2018, Applied Sciences.

[89]  David Isele,et al.  Navigating Occluded Intersections with Autonomous Vehicles Using Deep Reinforcement Learning , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[90]  Xian-Sheng Hua,et al.  Multi-Task Vehicle Detection With Region-of-Interest Voting , 2018, IEEE Transactions on Image Processing.

[91]  Dimitar Filev,et al.  Advanced planning for autonomous vehicles using reinforcement learning and deep inverse reinforcement learning , 2019, Robotics Auton. Syst..

[92]  Samira Chouraqui,et al.  Novel SMC control design for path following of autonomous vehicles with uncertainties and mismatched disturbances , 2018, International Journal of Dynamics and Control.

[93]  Alan Wee-Chung Liew,et al.  A Likelihood-Based Data Fusion Model for the Integration of Multiple Sensor Data: A Case Study with Vision and Lidar Sensors , 2015, RiTA.

[94]  Matthew Derry,et al.  Challenges in Perception and Decision Making for Intelligent Automotive Vehicles: A Case Study , 2016, IEEE Transactions on Intelligent Vehicles.

[95]  Mark E. Campbell,et al.  Contingency Planning Over Probabilistic Obstacle Predictions for Autonomous Road Vehicles , 2013, IEEE Transactions on Robotics.

[96]  Yang Yang Ye,et al.  Lane detection method based on lane structural analysis and CNNs , 2018 .