Safe Reinforcement Learning with Probabilistic Control Barrier Functions for Ramp Merging

Prior work has looked at applying reinforcement learning and imitation learning approaches to autonomous driving scenarios, but either the safety or the efficiency of the algorithm is compromised. With the use of control barrier functions embedded into the reinforcement learning policy, we arrive at safe policies to optimize the performance of the autonomous driving vehicle. However, control barrier functions need a good approximation of the model of the car. We use probabilistic control barrier functions as an estimate of the model uncertainty. The algorithm is implemented as an online version in the CARLA (Dosovitskiy et al., 2017) Simulator and as an offline version on a dataset extracted from the NGSIM Database. The proposed algorithm is not just a safe ramp merging algorithm but a safe autonomous driving algorithm applied to address ramp merging on highways.

[1]  Samir Aknine,et al.  Novel Decision-Making Strategy for Connected and Autonomous Vehicles in Highway On-Ramp Merging , 2022, IEEE Transactions on Intelligent Transportation Systems.

[2]  Nikolai Matni,et al.  Learning Robust Output Control Barrier Functions from Safe Expert Demonstrations , 2021, ArXiv.

[3]  John M. Dolan,et al.  Probabilistic Safety-Assured Adaptive Merging Control for Autonomous Vehicles , 2021, 2021 IEEE International Conference on Robotics and Automation (ICRA).

[4]  Shengbo Eben Li,et al.  Model-based Constrained Reinforcement Learning using Generalized Control Barrier Function , 2021, 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[5]  Shengbo Eben Li,et al.  Feasibility Enhancement of Constrained Receding Horizon Control Using Generalized Control Barrier Function , 2021, 2021 4th IEEE International Conference on Industrial Cyber-Physical Systems (ICPS).

[6]  I. Tasic,et al.  Safety analysis of freeway on-ramp merging with the presence of autonomous vehicles. , 2021, Accident; analysis and prevention.

[7]  Sifa Zheng,et al.  Numerically Stable Dynamic Bicycle Model for Discrete-time Control , 2020, 2021 IEEE Intelligent Vehicles Symposium Workshops (IV Workshops).

[8]  Koushil Sreenath,et al.  Safety-Critical Model Predictive Control with Discrete-Time Control Barrier Function , 2020, 2021 American Control Conference (ACC).

[9]  Magnus Egerstedt,et al.  Enhancing Game-Theoretic Autonomous Car Racing Using Control Barrier Functions , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[10]  Jason J. Choi,et al.  Reinforcement Learning for Safety-Critical Control under Model Uncertainty, using Control Lyapunov Functions and Control Barrier Functions , 2020, Robotics: Science and Systems.

[11]  Ashish Kapoor,et al.  Multi-Robot Collision Avoidance under Uncertainty with Probabilistic Safety Barrier Certificates , 2019, NeurIPS.

[12]  M. Franceschetti,et al.  Probabilistic Safety Constraints for Learned High Relative Degree System Dynamics , 2019, L4DC.

[13]  Tong Duy Son,et al.  Safety-Critical Control for Non-affine Nonlinear Systems with Application on Autonomous Vehicle , 2019, 2019 IEEE 58th Conference on Decision and Control (CDC).

[14]  S. Li,et al.  Adaptive dynamic programming for nonaffine nonlinear optimal control problem with state constraints , 2019, Neurocomputing.

[15]  Aaron D. Ames,et al.  Adaptive Safety with Control Barrier Functions , 2019, 2020 American Control Conference (ACC).

[16]  Andrew Clark,et al.  Control Barrier Functions for Complete and Incomplete Information Stochastic Systems , 2019, 2019 American Control Conference (ACC).

[17]  Sergey Levine,et al.  When to Trust Your Model: Model-Based Policy Optimization , 2019, NeurIPS.

[18]  Paulo Tabuada,et al.  Control Barrier Functions: Theory and Applications , 2019, 2019 18th European Control Conference (ECC).

[19]  V. Koltun,et al.  CARLA: An Open Urban Driving Simulator , 2017, CoRL.

[20]  Henry Prakken,et al.  On the problem of making autonomous vehicles conform to traffic law , 2017, Artificial Intelligence and Law.

[21]  John M. Dolan,et al.  Intention estimation for ramp merging control in autonomous driving , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[22]  Pieter Abbeel,et al.  Constrained Policy Optimization , 2017, ICML.

[23]  Aaron D. Ames,et al.  Safety Barrier Certificates for Collisions-Free Multirobot Systems , 2017, IEEE Transactions on Robotics.

[24]  Najah AbuAli,et al.  Driver Behavior Modeling: Developments and Future Directions , 2016 .

[25]  Paulo Tabuada,et al.  Control Barrier Function Based Quadratic Programs for Safety Critical Systems , 2016, IEEE Transactions on Automatic Control.

[26]  Katia P. Sycara,et al.  Distributed dynamic priority assignment and motion planning for multiple mobile robots with kinodynamic constraints , 2016, 2016 American Control Conference (ACC).

[27]  Koushil Sreenath,et al.  Exponential Control Barrier Functions for enforcing high relative-degree safety-critical constraints , 2016, 2016 American Control Conference (ACC).

[28]  Jin-Woo Lee,et al.  Tunable and stable real-time trajectory planning for urban autonomous driving , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[29]  Paulo Tabuada,et al.  Control barrier function based quadratic programs with application to adaptive cruise control , 2014, 53rd IEEE Conference on Decision and Control.

[30]  Masahiro Ono,et al.  Chance-Constrained Optimal Path Planning With Obstacles , 2011, IEEE Transactions on Robotics.

[31]  Stephen P. Boyd,et al.  Convex Optimization , 2004, IEEE Transactions on Automatic Control.

[32]  Soyoung Ahn,et al.  Freeway traffic oscillations : microscopic analysis of formations and propagations using Wavelet Transform , 2011 .