Proving Ground Test of a DDPG-based Vehicle Trajectory Planner

The paper presents real-world test cases of an optimal trajectory design solution that combines modern control techniques with machine learning. The first step of the current research is to train a reinforcement learning agent in a simulated environment, where the conditions and the applied vehicle are modeled. System dynamics is described by a nonlinear single-track vehicle with dynamic wheel model. The designed trajectory is evaluated by driving the vehicle using a control loop. The reward of the method is based on the sum of different measures considering safety and passenger comfort. The proposed method forms a special one-step reinforcement learning task handled by Deep Deterministic Policy Gradient (DDPG) learning agent. As a result, the learning process provides a real-time neural-network-based motion planner and a tracking algorithm. The evaluation of the algorithm under real conditions is made by using an experimental test vehicle. The test setup contains a high precision GPS module, an automotive inertial sensor, an industrial PC, and communication interface devices. The test cases were performed on the ZalaZone automotive proving ground.

[1]  Junjie Wang,et al.  Lane Change Decision-making through Deep Reinforcement Learning with Rule-based Constraints , 2019, 2019 International Joint Conference on Neural Networks (IJCNN).

[2]  Hans B. Pacejka.  Chapter 8 – Applications of transient tyre models , 2006 .

[3]  Johann Marius Zöllner,et al.  Learning how to drive in a real world simulation with deep Q-Networks , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[4]  Xin Zhang,et al.  End to End Learning for Self-Driving Cars , 2016, ArXiv.

[5]  Riccardo Marino,et al.  Nested PID steering control for lane keeping in autonomous vehicles , 2011 .

[6]  Zsolt Szalay,et al.  A Multi-layer Autonomous Vehicle and Simulation Validation Ecosystem Axis: ZalaZONE , 2018, IAS.

[7]  Qichao Zhang,et al.  Reinforcement Learning and Deep Learning Based Lateral Control for Autonomous Driving [Application Notes] , 2019, IEEE Comput. Intell. Mag..

[8]  Jae-il Jung,et al.  Decision-Making System for Lane Change Using Deep Reinforcement Learning in Connected and Automated Driving , 2019, Electronics.

[9]  Tamas Becsi,et al.  Q-learning based Reinforcement Learning Approach for Lane Keeping , 2018, 2018 IEEE 18th International Symposium on Computational Intelligence and Informatics (CINTI).

[10]  Tamás Bécsi,et al.  Model Based Trajectory Planning for Highly Automated Road Vehicles , 2017 .

[11]  Tamás Bécsi,et al.  Hybrid Trajectory Planning for Autonomous Vehicles using Neural Networks , 2018, 2018 IEEE 18th International Symposium on Computational Intelligence and Informatics (CINTI).

[12]  István Varga,et al.  Development of a Test Track for Driverless Cars: Vehicle Design, Track Configuration, and Liability Considerations , 2017 .

[13]  Xin Li,et al.  Reinforcement learning based overtaking decision-making for highway autonomous driving , 2015, 2015 Sixth International Conference on Intelligent Control and Information Processing (ICICIP).

[14]  Sebastian Thrun,et al.  Stanley: The robot that won the DARPA Grand Challenge , 2006, J. Field Robotics.

[15]  Emilio Frazzoli,et al.  A Survey of Motion Planning and Control Techniques for Self-Driving Urban Vehicles , 2016, IEEE Transactions on Intelligent Vehicles.

[16]  Katherine Rose Driggs-Campbell,et al.  Improved Robustness and Safety for Autonomous Vehicle Control with Adversarial Reinforcement Learning , 2018, 2018 IEEE Intelligent Vehicles Symposium (IV).

[17]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[18]  Xin Li,et al.  A Reinforcement Learning Approach to Autonomous Decision Making of Intelligent Vehicles on Highways , 2020, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[19]  Szil'ard Aradi,et al.  Survey of Deep Reinforcement Learning for Motion Planning of Autonomous Vehicles , 2020, IEEE Transactions on Intelligent Transportation Systems.

[20]  Zhiheng Li,et al.  Cooperative Lane Changing via Deep Reinforcement Learning , 2019, ArXiv.

[21]  Péter Gáspár,et al.  Hybrid DDPG Approach for Vehicle Motion Planning , 2019, ICINCO.

[22]  David González,et al.  A Review of Motion Planning Techniques for Automated Vehicles , 2016, IEEE Transactions on Intelligent Transportation Systems.

[23]  Carl-Johan Hoel,et al.  Automated Speed and Lane Change Decision Making using Deep Reinforcement Learning , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[24]  Yuan Zhu,et al.  Deep Q-Network Based Decision Making for Autonomous Driving , 2019, 2019 3rd International Conference on Robotics and Automation Sciences (ICRAS).