A Reinforcement Learning Based Approach for Automated Lane Change Maneuvers

Lane change is a crucial vehicle maneuver which needs coordination with surrounding vehicles. Automated lane changing functions built on rule-based models may perform well under pre-defined operating conditions, but they may be prone to failure when unexpected situations are encountered. In our study, we proposed a Reinforcement Learning based approach to train the vehicle agent to learn an automated lane change behavior such that it can intelligently make a lane change under diverse and even unforeseen scenarios. Particularly, we treated both state space and action space as continuous, and designed a Q-function approximator that has a closed-form greedy policy, which contributes to the computation efficiency of our deep Q-learning algorithm. Extensive simulations are conducted for training the algorithm, and the results illustrate that the Reinforcement Learning based vehicle agent is capable of learning a smooth and efficient driving policy for lane change maneuvers.

[1]  Sergey Levine,et al.  Trust Region Policy Optimization , 2015, ICML.

[2]  Peter Hidas,et al.  Modelling vehicle interactions in microscopic simulation of merging and weaving , 2005 .

[3]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[4]  Yishay Mansour,et al.  Policy Gradient Methods for Reinforcement Learning with Function Approximation , 1999, NIPS.

[5]  Sergey Levine,et al.  High-Dimensional Continuous Control Using Generalized Advantage Estimation , 2015, ICLR.

[6]  Wang Bing,et al.  An Adaptive PID Controller with Neural Network Self-Tuning for Vehicle Lane Keeping System , 2009 .

[7]  Robert Babuska,et al.  Experience Replay for Real-Time Reinforcement Learning Control , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[8]  Sergey Levine,et al.  Continuous Deep Q-Learning with Model-based Acceleration , 2016, ICML.

[9]  Shannon Hetrick,et al.  Examination of Driver Lane Change Behavior and the Potential Effectiveness of Warning Onset Rules for Lane Change or "Side" Crash Avoidance Systems , 1997 .

[10]  Rishi Bedi,et al.  Deep Reinforcement Learning for Simulated Autonomous Vehicle Control , 2016 .

[11]  Fabien Moutarde,et al.  A hierarchical Model Predictive Control framework for on-road formation control of autonomous vehicles , 2016, 2016 IEEE Intelligent Vehicles Symposium (IV).

[12]  P. G. Gipps,et al.  A MODEL FOR THE STRUCTURE OF LANE-CHANGING DECISIONS , 1986 .

[13]  Francesco Borrelli,et al.  A machine learning approach for personalized autonomous lane change initiation and control , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[14]  Helbing,et al.  Congested traffic states in empirical observations and microscopic simulations , 2000, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[15]  Patrick M. Pilarski,et al.  Model-Free reinforcement learning with continuous action in practice , 2012, 2012 American Control Conference (ACC).

[16]  Ching-Yao Chan,et al.  Formulation of deep reinforcement learning architecture toward autonomous driving for on-ramp merge , 2017, 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC).

[17]  Jung-Ha Kim,et al.  Lane change and path planning of autonomous vehicles using GIS , 2015, 2015 12th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI).

[18]  A. B. Rad,et al.  Lane Change Algorithm for Autonomous Vehicles via Virtual Curvature Method , 2009 .

[19]  Etienne Perot,et al.  End-to-End Deep Reinforcement Learning for Lane Keeping Assist , 2016, ArXiv.