Event-Triggered Model Predictive Control with Deep Reinforcement Learning for Autonomous Driving

—Event-triggered model predictive control (eMPC) is a popular optimal control method with an aim to alleviate the computation and/or communication burden of MPC. However, it generally requires priori knowledge of the closed-loop system behavior along with the communication characteristics for de- signing the event-trigger policy. This paper attempts to solve this challenge by proposing an efficient eMPC framework and demonstrate successful implementation of this framework on the autonomous vehicle path following. First of all, a model-free rein- forcement learning (RL) agent is used to learn the optimal event-trigger policy without the need for a complete dynamical system and communication knowledge in this framework. Furthermore, techniques including prioritized experience replay (PER) buffer and long-short term memory (LSTM) are employed to foster exploration and improve training efficiency. In this paper, we use the proposed framework with three deep RL algorithms, i.e., Double Q-learning (DDQN), Proximal Policy Optimization (PPO), and Soft Actor-Critic (SAC), to solve this problem. Experimental results show that all three deep RL-based eMPC (deep-RL-eMPC) can achieve better evaluation performance than the conventional threshold-based and previous linear Q-based approach in the autonomous path following. In particular, PPO-eMPC with LSTM and DDQN-eMPC with PER and LSTM obtains a superior balance between the closed-loop control performance and event-trigger frequency. The associated code is open-sourced and available at: https://github.com/DangFengying/ RL-based-event-triggered-MPC.

[1]  Xiangyu Meng,et al.  Reinforcement Learning-based Event-Triggered Model Predictive Control for Autonomous Vehicle Path Following , 2022, 2022 American Control Conference (ACC).

[2]  Jun Chen,et al.  Event-Triggered Model Predictive Control for Autonomous Vehicle with Rear Steering , 2022, SAE Technical Paper Series.

[3]  Zonggen Yi,et al.  Comparison of Event-Triggered Model Predictive Control for Autonomous Vehicle Path Tracking , 2021, 2021 IEEE Conference on Control Technology and Applications (CCTA).

[4]  Runze Yang,et al.  A Review on the Application of the MPC Technology in Wind Power Control of Wind Farms , 2021, Journal of Energy and Power Technology.

[5]  Hiroyuki Okuda,et al.  Real-Time Implementation of Randomized Model Predictive Control for Autonomous Driving , 2021, IEEE Transactions on Intelligent Vehicles.

[6]  Tor Arne Johansen,et al.  Optimization of the Model Predictive Control Update Interval Using Reinforcement Learning , 2020, IFAC-PapersOnLine.

[7]  Md. Noor-A.-Rahim,et al.  Machine Learning in Event-Triggered Control: Recent Advances and Open Issues , 2020, IEEE Access.

[8]  Takamitsu Matsubara,et al.  Autonomous boat driving system using sample‐efficient model predictive control‐based reinforcement learning approach , 2020, J. Field Robotics.

[9]  Yoshua Bengio,et al.  Revisiting Fundamentals of Experience Replay , 2020, ICML.

[10]  Fabrizio Dabbene,et al.  Computationally efficient stochastic MPC: a probabilistic scaling approach , 2020, 2020 IEEE Conference on Control Technology and Applications (CCTA).

[11]  Takamitsu Matsubara,et al.  Sample-and-computation-efficient Probabilistic Model Predictive Control with Random Features , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[12]  M. Hutter,et al.  Practical Reinforcement Learning For MPC: Learning from sparse objectives in under an hour on a real robot , 2020, L4DC.

[13]  Takamitsu Matsubara,et al.  Reinforcement Learning Boat Autopilot: A Sample-efficient and Model Predictive Control based Approach , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[14]  Petros Christodoulou,et al.  Soft Actor-Critic for Discrete Action Settings , 2019, ArXiv.

[15]  Marco Hutter,et al.  Deep Value Model Predictive Control , 2019, CoRL.

[16]  Marco Hutter,et al.  Data-Driven Model Predictive Control for Trajectory Tracking With a Robotic Arm , 2019, IEEE Robotics and Automation Letters.

[17]  Alexander Liniger,et al.  Learning-Based Model Predictive Control for Autonomous Racing , 2019, IEEE Robotics and Automation Letters.

[18]  Karl Henrik Johansson,et al.  Event-Triggered Model Predictive Control With a Statistical Learning , 2019, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[19]  Cheng Liu,et al.  Computationally efficient MPC for path following of underactuated marine vessels using projection neural network , 2019, Neural Computing and Applications.

[20]  Karl Henrik Johansson,et al.  Event-triggered Pulse Control with Model Learning (if Necessary) , 2019, 2019 American Control Conference (ACC).

[21]  Jacob Rosen,et al.  Autonomous Tissue Manipulation via Surgical Robot Using Learning Based Model Predictive Control , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[22]  Sebastian Trimpe,et al.  Deep Reinforcement Learning for Event-Triggered Control , 2018, 2018 IEEE Conference on Decision and Control (CDC).

[23]  Daoliang Li,et al.  Model predictive control and its application in agriculture: A review , 2018, Comput. Electron. Agric..

[24]  Ryad Chellali,et al.  An Overview of Nature-Inspired, Conventional, and Hybrid Methods of Autonomous Vehicle Path Planning , 2018, Journal of Advanced Transportation.

[25]  Alberto Bemporad,et al.  Model Predictive Control (MPC) for Enhancing Building and HVAC System Energy Efficiency: Problem Formulation, Applications and Opportunities , 2018 .

[26]  Sergey Levine,et al.  Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor , 2018, ICML.

[27]  Alec Radford,et al.  Proximal Policy Optimization Algorithms , 2017, ArXiv.

[28]  Marc Peter Deisenroth,et al.  Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Control , 2017, AISTATS.

[29]  Chang Liu,et al.  Path planning for autonomous vehicles using model predictive control , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[30]  Juraj Kabzan,et al.  Cautious Model Predictive Control Using Gaussian Process Regression , 2017, IEEE Transactions on Control Systems Technology.

[31]  W. P. M. H. Heemels,et al.  Robust Event-Triggered MPC With Guaranteed Asymptotic Bound and Average Sampling Rate , 2017, IEEE Transactions on Automatic Control.

[32]  Gerald O'Regan,et al.  Introduction to the History of Computing: A Computing History Primer , 2016 .

[33]  Tom Schaul,et al.  Prioritized Experience Replay , 2015, ICLR.

[34]  Dan Xu,et al.  ACIS: An Improved Actor-Critic Method for POMDPs with Internal State , 2015, 2015 IEEE 27th International Conference on Tools with Artificial Intelligence (ICTAI).

[35]  Ning He,et al.  Event-Based Robust Sampled-Data Model Predictive Control: A Non-Monotonic Lyapunov Function Approach , 2015, IEEE Transactions on Circuits and Systems I: Regular Papers.

[36]  Francesco Borrelli,et al.  Kinematic and dynamic vehicle models for autonomous driving control design , 2015, 2015 IEEE Intelligent Vehicles Symposium (IV).

[37]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[38]  Michael I. Jordan,et al.  Trust Region Policy Optimization , 2015, ICML.

[39]  Huiping Li,et al.  Event-triggered robust model predictive control of continuous-time nonlinear systems , 2014, Autom..

[40]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[41]  Dimos V. Dimarogonas,et al.  Novel event-triggered strategies for Model Predictive Controllers , 2011, IEEE Conference on Decision and Control and European Control Conference.

[42]  Sebastian Thrun,et al.  Path Planning for Autonomous Vehicles in Unknown Semi-structured Environments , 2010, Int. J. Robotics Res..

[43]  M. Soroush,et al.  Model Predictive Control Tuning Methods: A Review , 2010 .

[44]  Rajesh Rajamani,et al.  Vehicle dynamics and control , 2005 .

[45]  Peter L. Bartlett,et al.  Infinite-Horizon Policy-Gradient Estimation , 2001, J. Artif. Intell. Res..

[46]  S. Hochreiter,et al.  Long Short-Term Memory , 1997, Neural Computation.

[47]  Long Ji Lin,et al.  Self-improving reactive agents based on reinforcement learning, planning and teaching , 1992, Machine Learning.

[48]  Munther A. Dahleh,et al.  Event-Triggered Reinforcement Learning; An Application to Buildings' Micro-Climate Control , 2020, AAAI Spring Symposium: MLPS.

[49]  Frank Allgöwer,et al.  Learning-Based Robust Model Predictive Control with State-Dependent Uncertainty , 2018 .

[50]  Angela P. Schoellig,et al.  Learning‐based Nonlinear Model Predictive Control to Improve Vision‐based Mobile Robot Path Tracking , 2016, J. Field Robotics.

[51]  Longxin Lin Self-Improving Reactive Agents Based on Reinforcement Learning, Planning and Teaching , 2004, Machine Learning.