Automated Speed and Lane Change Decision Making using Deep Reinforcement Learning

This paper introduces a method, based on deep reinforcement learning, for automatically generating a general purpose decision making function. A Deep Q-Network agent was trained in a simulated environment to handle speed and lane change decisions for a truck-trailer combination. In a highway driving case, it is shown that the method produced an agent that matched or surpassed the performance of a commonly used reference model. To demonstrate the generality of the method, the exact same algorithm was also tested by training it for an overtaking case on a road with oncoming traffic. Furthermore, a novel way of applying a convolutional neural network to high level input that represents interchangeable objects is also introduced.

[1]  Helbing,et al.  Congested traffic states in empirical observations and microscopic simulations , 2000, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[2]  Julian Eggert,et al.  Complex Lane Change Behavior in the Foresighted Driver Model , 2015, 2015 IEEE 18th International Conference on Intelligent Transportation Systems.

[3]  Amnon Shashua,et al.  Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving , 2016, ArXiv.

[4]  Carl-Johan Hoel,et al.  An Evolutionary Approach to General-Purpose Automated Speed and Lane Change Behavior , 2017, 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA).

[5]  Eduardo F. Morales,et al.  An Introduction to Reinforcement Learning , 2011 .

[6]  Daniel Bartz,et al.  Truck Automation: Testing and Trusting the Virtual Driver , 2016 .

[7]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[8]  Mykel J. Kochenderfer,et al.  The value of inferring the internal state of traffic participants for autonomous freeway driving , 2017, 2017 American Control Conference (ACC).

[9]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[10]  Dirk Helbing,et al.  General Lane-Changing Model MOBIL for Car-Following Models , 2007 .

[11]  Markus Maurer,et al.  Towards Tactical Lane Change Behavior Planning for Automated Vehicles , 2015, 2015 IEEE 18th International Conference on Intelligent Transportation Systems.

[12]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[13]  Demis Hassabis,et al.  Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm , 2017, ArXiv.

[14]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[15]  K. Ahmed.  Modeling drivers' acceleration and lane changing behavior , 1999 .

[16]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[17]  Daniel J. Fagnant,et al.  Preparing a Nation for Autonomous Vehicles: Opportunities, Barriers and Policy Recommendations , 2015 .

[18]  Leslie Pack Kaelbling,et al.  Planning and Acting in Partially Observable Stochastic Domains , 1998, Artif. Intell..

[19]  Rob Gray,et al.  A Two-Point Visual Control Model of Steering , 2004, Perception.

[20]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[21]  Tom Schaul,et al.  Prioritized Experience Replay , 2015, ICLR.

[22]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[23]  P. G. Gipps,et al.  A MODEL FOR THE STRUCTURE OF LANE-CHANGING DECISIONS , 1986 .

[24]  Jonas Fredriksson,et al.  If, When, and How to Perform Lane Change Maneuvers on Highways , 2016, IEEE Intelligent Transportation Systems Magazine.

[25]  Demis Hassabis,et al.  Mastering the game of Go without human knowledge , 2017, Nature.

[26]  David Silver,et al.  Deep Reinforcement Learning with Double Q-Learning , 2015, AAAI.

[27]  Etienne Perot,et al.  Deep Reinforcement Learning framework for Autonomous Driving , 2017, Autonomous Vehicles and Machines.