Reinforcement Learning-based Path Following Control for a Vehicle with Variable Delay in the Drivetrain

In this contribution we propose a reinforcement learning-based controller able to solve the path following problem for vehicles with significant delay in the drivetrain. To efficiently train the controller, a control-oriented simulation model for a vehicle with combustion engine, automatic gear box and hydraulic brake system has been developed. In addition, to enhance the reinforcement learning-based controller, we have incorporated preview information in the feedback state to better deal with the delays. We present our approach of designing a reward function which enables the reinforcement learning-based controller to solve the problem. The controller is trained using the Soft Actor-Critic algorithm by incorporating the developed simulation model. Finally, the performance and robustness is evaluated in simulation. Our controller is able to follow an unseen path and is robust against variations in the vehicle parameters, in our case an additional payload.

[1]  Martin Lauer,et al.  Learning Path Tracking for Real Car-like Mobile Robots From Simulation , 2019, 2019 European Conference on Mobile Robots (ECMR).

[2]  J. Andrew Bagnell,et al.  Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy , 2010 .

[3]  Jonathan Brembeck,et al.  Model Based Energy Management and State Estimation for the Robotic Electric Vehicle ROboMObil , 2018 .

[4]  Azer Bestavros,et al.  Reinforcement Learning for UAV Attitude Control , 2018, ACM Trans. Cyber Phys. Syst..

[5]  Alois Knoll,et al.  Deep Reinforcement Learning for Predictive Longitudinal Control of Automated Vehicles , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[6]  Fabien Moutarde,et al.  Coupled Longitudinal and Lateral Control of a Vehicle using Deep Learning , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[7]  Atil Iscen,et al.  Sim-to-Real: Learning Agile Locomotion For Quadruped Robots , 2018, Robotics: Science and Systems.

[8]  Jonathan Brembeck,et al.  Nonlinear Constrained Moving Horizon Estimation Applied to Vehicle Position Estimation , 2019, Sensors.

[9]  Johann Bals,et al.  ROMO - THE ROBOTIC ELECTRIC VEHICLE , 2011 .

[10]  Jonathan Brembeck,et al.  Experimental investigation of online path planning for electric vehicles , 2016, 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC).

[11]  D. Hrovat,et al.  Hydraulic brake system modeling and control for active control of vehicle dynamics , 1999, Proceedings of the 1999 American Control Conference (Cat. No. 99CH36251).

[12]  Georg Bednarek,et al.  Achtgang-Automatikgetriebe für den neuen Opel Insignia , 2016 .

[13]  Andreas Witt,et al.  Die neuen Vier-und Sechszylinder-Ottomotoren von BMW mit Schichtbrennverfahren , 2007 .

[14]  David Janz,et al.  Learning to Drive in a Day , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[15]  Dirk Zimmer.  A Planar Mechanical Library for Teaching Modelica , 2012 .

[16]  Sergey Levine,et al.  Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor , 2018, ICML.

[17]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[18]  Gyula Mester,et al.  Motion Control of Wheeled Mobile Robots , 2006 .

[19]  Jonathan Brembeck,et al.  Advanced path following control of an overactuated robotic vehicle , 2015, 2015 IEEE Intelligent Vehicles Symposium (IV).

[20]  Henry Zhu,et al.  Soft Actor-Critic Algorithms and Applications , 2018, ArXiv.

[21]  Anastasios M. Lekkas,et al.  Curved Path Following with Deep Reinforcement Learning: Results from Three Vessel Models , 2018, OCEANS 2018 MTS/IEEE Charleston.