Longitudinal Dynamic versus Kinematic Models for Car-Following Control Using Deep Reinforcement Learning

The majority of current studies on autonomous vehicle control via deep reinforcement learning (DRL) utilize point-mass kinematic models, neglecting vehicle dynamics which includes acceleration delay and acceleration command dynamics. The acceleration delay, which results from sensing and actuation delays, results in delayed execution of the control inputs. The acceleration command dynamics dictates that the actual vehicle acceleration does not rise up to the desired command acceleration instantaneously due to dynamics. In this work, we investigate the feasibility of applying DRL controllers trained using vehicle kinematic models to more realistic driving control with vehicle dynamics. We consider a particular longitudinal car-following control, i.e., Adaptive Cruise Control (ACC), problem solved via DRL using a point-mass kinematic model. When such a controller is applied to car following with vehicle dynamics, we observe significantly degraded car-following performance. Therefore, we redesign the DRL framework to accommodate the acceleration delay and acceleration command dynamics by adding the delayed control inputs and the actual vehicle acceleration to the reinforcement learning environment state, respectively. The training results show that the redesigned DRL controller results in near-optimal control performance of car following with vehicle dynamics considered when compared with dynamic programming solutions.

[1]  Martijn Wisse,et al.  The design of LEO: A 2D bipedal walking robot for online autonomous Reinforcement Learning , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[2]  Reza N. Jazar,et al.  Vehicle Dynamics: Theory and Application , 2009 .

[3]  Bin Wang,et al.  A supervised Actor–Critic approach for adaptive cruise control , 2013, Soft Comput..

[4]  John M. Dolan,et al.  Automatically Generated Curriculum based Reinforcement Learning for Autonomous Vehicles in Urban Environment , 2018, 2018 IEEE Intelligent Vehicles Symposium (IV).

[5]  Robert Babuska,et al.  Control delay in Reinforcement Learning for real-time dynamic systems: A memoryless approach , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[6]  Florian Kuhnt,et al.  Adaptive Behavior Generation for Autonomous Driving using Deep Reinforcement Learning with Compact Semantic States , 2018, 2018 IEEE Intelligent Vehicles Symposium (IV).

[7]  Meixin Zhu,et al.  Human-Like Autonomous Car-Following Model with Deep Reinforcement Learning , 2018, Transportation Research Part C: Emerging Technologies.

[8]  Krzysztof Czarnecki,et al.  Urban Driving with Multi-Objective Deep Reinforcement Learning , 2018, AAMAS.

[9]  D. Naidu,et al.  Optimal Control Systems , 2018 .

[10]  Tony Givargis,et al.  Fine-grained acceleration control for autonomous intersection management using deep reinforcement learning , 2017, 2017 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computed, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI).

[11]  Péter Gáspár,et al.  Policy Gradient Based Reinforcement Learning Approach for Autonomous Highway Driving , 2018, 2018 IEEE Conference on Control Technology and Applications (CCTA).

[12]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[13]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[14]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[15]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[16]  Konstantinos V. Katsikopoulos,et al.  Markov decision processes with delays and asynchronous cost collection , 2003, IEEE Trans. Autom. Control..

[17]  John M. Dolan,et al.  POMDP and Hierarchical Options MDP with Continuous Actions for Autonomous Driving at Intersections , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[18]  Matthew W. Hoffman,et al.  Distributed Distributional Deterministic Policy Gradients , 2018, ICLR.

[19]  Robert Babuška,et al.  On-line Reinforcement Learning for Nonlinear Motion Control: Quadratic and Non-Quadratic Reward Functions , 2014 .

[20]  David Silver,et al.  Deep Reinforcement Learning with Double Q-Learning , 2015, AAAI.

[21]  Charles Desjardins,et al.  Cooperative Adaptive Cruise Control: A Reinforcement Learning Approach , 2011, IEEE Transactions on Intelligent Transportation Systems.

[22]  Wojciech Zaremba,et al.  OpenAI Gym , 2016, ArXiv.

[23]  Thomas J. Walsh,et al.  Learning and planning in environments with delayed feedback , 2009, Autonomous Agents and Multi-Agent Systems.

[24]  Ching-Yao Chan,et al.  Autonomous Ramp Merge Maneuver Based on Reinforcement Learning with Continuous Action Space , 2018, ArXiv.

[25]  Ching-Yao Chan,et al.  A Reinforcement Learning Based Approach for Automated Lane Change Maneuvers , 2018, 2018 IEEE Intelligent Vehicles Symposium (IV).

[26]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[27]  Nathan van de Wouw,et al.  Design and experimental evaluation of cooperative adaptive cruise control , 2011, 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC).

[28]  Sergey Levine,et al.  Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor , 2018, ICML.

[29]  David Isele,et al.  Navigating Occluded Intersections with Autonomous Vehicles Using Deep Reinforcement Learning , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[30]  Peter Stone,et al.  TEXPLORE: real-time sample-efficient reinforcement learning for robots , 2012, Machine Learning.

[31]  Ulf Holmberg,et al.  A Modular CACC System Integration and Design , 2012, IEEE Transactions on Intelligent Transportation Systems.

[32]  William Whittaker,et al.  Autonomous driving in urban environments: Boss and the Urban Challenge , 2008, J. Field Robotics.

[33]  Azim Eskandarian,et al.  Handbook of Intelligent Vehicles , 2012 .

[34]  Alois Knoll,et al.  Deep Reinforcement Learning for Predictive Longitudinal Control of Automated Vehicles , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[35]  Tao Zhang,et al.  Design and Experimental Validation of a Cooperative Adaptive Cruise Control System Based on Supervised Reinforcement Learning , 2018, Applied Sciences.

[36]  David Silver,et al.  Learning values across many orders of magnitude , 2016, NIPS.