Hierarchical Reinforcement Learning Combined with Motion Primitives for Automated Overtaking

This paper presents a novel hierarchical reinforcement learning (HRL) framework for automated overtaking. The proposed framework is developed based on the semi-Markov decision process (SMDP) and motion primitives (MPs) which can be applied to different overtaking phases. Unlike the high-level decision and low-level control which are usually independent with each other, the high-level decision making and low-level control are combined by defining MPs with different time intervals. As for the high-level decision making, a SMDP Q-learning algorithm is adopted to realize decision-making of MPs. Besides, a development method of MPs used in the low-level control of automated overtaking is proposed. The performance of the HRL framework is tested in the simulation environment built in a driving simulator called CARLA. The results show that the HRL framework can determine the optimal trajectory under different driving styles of the overtaken vehicle.

[1]  K. Madhava Krishna,et al.  Overtaking Maneuvers in Simulated Highway Driving using Deep Reinforcement Learning , 2018, 2018 IEEE Intelligent Vehicles Symposium (IV).

[2]  Xiaofei Pei,et al.  Decision-Making for Oncoming Traffic Overtaking Scenario using Double DQN , 2019, 2019 3rd Conference on Vehicle Control and Intelligence (CVCI).

[3]  Wassim Najm,et al.  CRASH PROBLEM CHARACTERISTICS FOR THE INTELLIGENT VEHICLE INITIATIVE , 2001 .

[4]  Guodong Yin,et al.  Robust overtaking control of autonomous electric vehicle with parameter uncertainties , 2019, Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering.

[5]  Alonzo Kelly,et al.  Optimal Rough Terrain Trajectory Generation for Wheeled Mobile Robots , 2007, Int. J. Robotics Res..

[6]  Jianbo Lu,et al.  Autonomous Planning and Control for Intelligent Vehicles in Traffic , 2020, IEEE Transactions on Intelligent Transportation Systems.

[7]  Johan Karlsson,et al.  Temporal vs. spatial formulation of autonomous overtaking algorithms , 2016, 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC).

[8]  Sebastian Thrun,et al.  Towards fully autonomous driving: Systems and algorithms , 2011, 2011 IEEE Intelligent Vehicles Symposium (IV).

[9]  Doina Precup,et al.  Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning , 1999, Artif. Intell..

[10]  Germán Ros,et al.  CARLA: An Open Urban Driving Simulator , 2017, CoRL.

[11]  Carl-Johan Hoel,et al.  Automated Speed and Lane Change Decision Making using Deep Reinforcement Learning , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[12]  Sebastian Thrun,et al.  Stanley: The robot that won the DARPA Grand Challenge , 2006, J. Field Robotics.

[13]  Emilio Frazzoli,et al.  A Survey of Motion Planning and Control Techniques for Self-Driving Urban Vehicles , 2016, IEEE Transactions on Intelligent Vehicles.

[14]  Jie Huang,et al.  Reinforcement Learning for Ramp Control: An Analysis of Learning Parameters , 2016 .

[15]  Ulrich Konigorski,et al.  A Simulation-Based Reinforcement Learning Approach for Long-Term Maneuver Planning in Highway Traffic Scenarios , 2019 .

[16]  Salsabil Khan,et al.  Collision avoidance system proposed by a model using NRF24L01 and infrared sensor , 2018 .

[17]  Ching-Yao Chan,et al.  Automated Driving Maneuvers under Interactive Environment based on Deep Reinforcement Learning , 2018, 1803.09200.

[18]  Yun Li,et al.  Patents, software, and hardware for PID control: an overview and analysis of the current art , 2006, IEEE Control Systems.