Continuous Control in Car Simulator with Deep Reinforcement Learning

Deep reinforcement learning (DRL), which can be trained without abundant labeled data required in supervised learning, plays an important role in autonomous vehicle researches. According to action space, DRL can be further divided into two classes: discrete domain and continuous domain. In this work, we focus on continuous steering control since it's impossible to switch among different discrete steering values at short intervals in reality. We first define the steering smoothness to quantify the degree of continuity. Then we propose a new penalty in reward shaping. We carry experiments based on Deep Deterministic Policy Gradient (DDPG) and Asynchronous Advantage Actor Critic (A3C), which are the state of the art in continuous domain. Results show that the proposed penalty improves the steering smoothness with both algorithms.

[1]  Nidhi Kalra,et al.  Autonomous Vehicle Technology: A Guide for Policymakers , 2014 .

[2]  Yang Gao,et al.  End-to-End Learning of Driving Models from Large-Scale Video Datasets , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Christos Dimitrakakis,et al.  TORCS, The Open Racing Car Simulator , 2005 .

[4]  Fawzi Nashashibi,et al.  End-to-End Race Driving with Deep Reinforcement Learning , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[5]  Daniele Loiacono,et al.  Simulated Car Racing Championship: Competition Software Manual , 2013, ArXiv.

[6]  Aaas News,et al.  Book Reviews , 1893, Buffalo Medical and Surgical Journal.

[7]  Xin Zhang,et al.  End to End Learning for Self-Driving Cars , 2016, ArXiv.

[8]  Hermann Winner,et al.  Three Decades of Driver Assistance Systems: Review and Future Perspectives , 2014, IEEE Intelligent Transportation Systems Magazine.

[9]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[10]  Marc Peter Deisenroth,et al.  Deep Reinforcement Learning: A Brief Survey , 2017, IEEE Signal Processing Magazine.

[11]  R. Stephenson.  A and V , 1962, The British journal of ophthalmology.

[12]  Alex Graves,et al.  Asynchronous Methods for Deep Reinforcement Learning , 2016, ICML.

[13]  Dean Pomerleau,et al.  ALVINN, an autonomous land vehicle in a neural network , 2015 .

[14]  Jun Tan,et al.  Parameterized Batch Reinforcement Learning for Longitudinal Control of Autonomous Land Vehicles , 2019, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[15]  Etienne Perot,et al.  End-to-End Deep Reinforcement Learning for Lane Keeping Assist , 2016, ArXiv.

[16]  Yuxi Li,et al.  Deep Reinforcement Learning: An Overview , 2017, ArXiv.

[17]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[18]  Sebastian Thrun,et al.  Junior: The Stanford entry in the Urban Challenge , 2008, J. Field Robotics.

[19]  Andreas Geiger,et al.  Vision meets robotics: The KITTI dataset , 2013, Int. J. Robotics Res..

[20]  Ming Jin,et al.  Control-Theoretic Analysis of Smoothness for Stability-Certified Reinforcement Learning , 2018, 2018 IEEE Conference on Decision and Control (CDC).

[21]  Rishi Bedi,et al.  Deep Reinforcement Learning for Simulated Autonomous Vehicle Control , 2016 .