Human-like Autonomous Vehicle Speed Control by Deep Reinforcement Learning with Double Q-Learning

Autonomous driving has become a popular research project. How to control vehicle speed is a core problem in autonomous driving. Automatic decision-making approaches, such as reinforcement learning (RL), have been applied to control the vehicle speed. However, the popular Q-learning algorithm is unstable in some games in the Atari 2600 domain. In this paper, a reinforcement learning approach called Double Q-learning is used to control a vehicle’s speed based on the environment constructed by naturalistic driving data. Depending on the concept of the direct perception approach, we propose a new method called integrated perception approach to construct the environment. The input of the model is made up of high dimensional data including road information processed from the video data and the low dimensional data processed from the sensors. During experiment, compared with deep Q-learning algorithm, double deep Q-learning has improvements both in terms of value accuracy and policy quality. Our model’s score is 271.73% times that of deep Q-learning.

[1]  Dean Pomerleau,et al.  ALVINN, an autonomous land vehicle in a neural network , 2015 .

[2]  Anton van den Hengel,et al.  Pedestrian Detection with Spatially Pooled Features and Structured Ensemble Learning , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Jianxiong Xiao,et al.  DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[4]  Marc G. Bellemare,et al.  The Arcade Learning Environment: An Evaluation Platform for General Agents (Extended Abstract) , 2012, IJCAI.

[5]  Young-Woo Seo,et al.  Utilizing instantaneous driving direction for enhancing lane-marking detection , 2014, 2014 IEEE Intelligent Vehicles Symposium Proceedings.

[6]  Xin Zhang,et al.  End to End Learning for Self-Driving Cars , 2016, ArXiv.

[7]  Paulo Peixoto,et al.  3D Lidar-based static and moving obstacle detection in driving environments: An approach based on voxels and multi-region ground planes , 2016, Robotics Auton. Syst..

[8]  Macario Cordel,et al.  Convolutional neural network for vehicle detection in low resolution traffic videos , 2016, 2016 IEEE Region 10 Symposium (TENSYMP).

[9]  Dean A. Pomerleau,et al.  Neural Network Perception for Mobile Robot Guidance , 1993 .

[10]  Jürgen Schmidhuber,et al.  Evolving large-scale neural networks for vision-based reinforcement learning , 2013, GECCO '13.

[11]  Alex Graves,et al.  Asynchronous Methods for Deep Reinforcement Learning , 2016, ICML.

[12]  Johannes Stallkamp,et al.  Detection of traffic signs in real-world images: The German traffic sign detection benchmark , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[13]  Jaerock Kwon,et al.  Lane following and obstacle detection techniques in autonomous driving vehicles , 2016, 2016 IEEE International Conference on Electro Information Technology (EIT).

[14]  B. Schiele,et al.  How Far are We from Solving Pedestrian Detection? , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Tom Schaul,et al.  Dueling Network Architectures for Deep Reinforcement Learning , 2015, ICML.

[16]  Roberto Hirata,et al.  Car detection in sequences of images of urban environments using mixture of deformable part models , 2014, Pattern Recognit. Lett..

[17]  S. Ullman.  Against direct perception , 1980, Behavioral and Brain Sciences.

[18]  David Silver,et al.  Deep Reinforcement Learning with Double Q-Learning , 2015, AAAI.

[19]  Andreas Geiger,et al.  Vision meets robotics: The KITTI dataset , 2013, Int. J. Robotics Res..

[20]  Johann Marius Zöllner,et al.  DeepTLR: A single deep convolutional network for detection and classification of traffic lights , 2016, 2016 IEEE Intelligent Vehicles Symposium (IV).

[21]  Jürgen Schmidhuber,et al.  Evolving deep unsupervised convolutional networks for vision-based reinforcement learning , 2014, GECCO.

[22]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[23]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[24]  Tao Mei,et al.  Robust lane marking detection under different road conditions , 2013, 2013 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[25]  Hao Yu,et al.  Vision-Based Lane Marking Detection and Moving Vehicle Detection , 2016, 2016 8th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC).

[26]  Chris Urmson,et al.  Traffic light mapping and detection , 2011, 2011 IEEE International Conference on Robotics and Automation.

[27]  Cuneyt Akinlar,et al.  On circular traffic sign detection and recognition , 2016, Expert Syst. Appl..

[28]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[29]  Tom Schaul,et al.  Prioritized Experience Replay , 2015, ICLR.

[30]  Fernando A. Mujica,et al.  An Empirical Evaluation of Deep Learning on Highway Driving , 2015, ArXiv.