Learning how to drive in a real world simulation with deep Q-Networks

We present a reinforcement learning approach using Deep Q-Networks to steer a vehicle in a 3D physics simulation. Relying solely on camera image input the approach directly learns steering the vehicle in an end-to-end manner. The system is able to learn human driving behavior without the need of any labeled training data. An action-based reward function is proposed, which is motivated by a potential use in real world reinforcement learning scenarios. Compared to a naive distance-based reward function, it improves the overall driving behavior of the vehicle agent. The agent is even able to reach comparable to human driving performance on a previously unseen track in our simulation environment.

[1]  Klaus C. J. Dietmayer,et al.  Autonomous driving at Ulm University: A modular, robust, and sensor-independent fusion approach , 2015, 2015 IEEE Intelligent Vehicles Symposium (IV).

[2]  Johannes Stallkamp,et al.  Detection of traffic signs in real-world images: The German traffic sign detection benchmark , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[3]  Anton van den Hengel,et al.  Pedestrian Detection with Spatially Pooled Features and Structured Ensemble Learning , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Jaerock Kwon,et al.  Lane following and obstacle detection techniques in autonomous driving vehicles , 2016, 2016 IEEE International Conference on Electro Information Technology (EIT).

[5]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[6]  Roberto Hirata,et al.  Car detection in sequences of images of urban environments using mixture of deformable part models , 2014, Pattern Recognit. Lett..

[7]  Tom Schaul,et al.  Prioritized Experience Replay , 2015, ICLR.

[8]  Fernando A. Mujica,et al.  An Empirical Evaluation of Deep Learning on Highway Driving , 2015, ArXiv.

[9]  Ralf Kohlhaas,et al.  Simulation framework for the development of autonomous small scale vehicles , 2016, 2016 IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR).

[10]  Cuneyt Akinlar,et al.  On circular traffic sign detection and recognition , 2016, Expert Syst. Appl..

[11]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[12]  Morgan Quigley,et al.  ROS: an open-source Robot Operating System , 2009, ICRA 2009.

[13]  Jianxiong Xiao,et al.  DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[14]  Dean A. Pomerleau,et al.  Neural Network Perception for Mobile Robot Guidance , 1993 .

[15]  Alex Graves,et al.  Asynchronous Methods for Deep Reinforcement Learning , 2016, ICML.

[16]  Andrew Howard,et al.  Design and use paradigms for Gazebo, an open-source multi-robot simulator , 2004, 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566).

[17]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[18]  Christoph Stiller,et al.  Functional system architectures towards fully automated driving , 2016, 2016 IEEE Intelligent Vehicles Symposium (IV).

[19]  Xin Zhang,et al.  End to End Learning for Self-Driving Cars , 2016, ArXiv.

[20]  Marc G. Bellemare,et al.  The Arcade Learning Environment: An Evaluation Platform for General Agents , 2012, J. Artif. Intell. Res..

[21]  Young-Woo Seo,et al.  Utilizing instantaneous driving direction for enhancing lane-marking detection , 2014, 2014 IEEE Intelligent Vehicles Symposium Proceedings.

[22]  Macario Cordel,et al.  Convolutional neural network for vehicle detection in low resolution traffic videos , 2016, 2016 IEEE Region 10 Symposium (TENSYMP).

[23]  Jürgen Schmidhuber,et al.  Evolving deep unsupervised convolutional networks for vision-based reinforcement learning , 2014, GECCO.

[24]  David Silver,et al.  Deep Reinforcement Learning with Double Q-Learning , 2015, AAAI.

[25]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[26]  Jürgen Schmidhuber,et al.  Evolving large-scale neural networks for vision-based TORCS , 2013, FDG.

[27]  Paulo Peixoto,et al.  3D Lidar-based static and moving obstacle detection in driving environments: An approach based on voxels and multi-region ground planes , 2016, Robotics Auton. Syst..

[28]  Tao Mei,et al.  Robust lane marking detection under different road conditions , 2013, 2013 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[29]  Julius Ziegler,et al.  Lanelets: Efficient map representation for autonomous driving , 2014, 2014 IEEE Intelligent Vehicles Symposium Proceedings.

[30]  Hao Yu,et al.  Vision-Based Lane Marking Detection and Moving Vehicle Detection , 2016, 2016 8th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC).

[31]  B. Schiele,et al.  How Far are We from Solving Pedestrian Detection? , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Tom Schaul,et al.  Dueling Network Architectures for Deep Reinforcement Learning , 2015, ICML.

[33]  Chris Urmson,et al.  Traffic light mapping and detection , 2011, 2011 IEEE International Conference on Robotics and Automation.

[34]  Dean Pomerleau,et al.  ALVINN, an autonomous land vehicle in a neural network , 2015 .

[35]  Johann Marius Zöllner,et al.  DeepTLR: A single deep convolutional network for detection and classification of traffic lights , 2016, 2016 IEEE Intelligent Vehicles Symposium (IV).

[36]  Longxin Lin.  Self-Improving Reactive Agents Based on Reinforcement Learning, Planning and Teaching , 2004, Machine Learning.