Robust Learning-based Predictive Control for Constrained Nonlinear Systems

The integration of machine learning methods and Model Predictive Control (MPC) has received increasing attention in recent years. In general, learning-based predictive control (LPC) is promising to build data-driven models and solve the online optimization problem with lower computational costs. However, the robustness of LPC is difficult to be guaranteed since there will be uncertainties due to function approximation used in machine learning algorithms. In this paper, a novel robust learning-based predictive control (r-LPC) scheme is proposed for constrained nonlinear systems with unknown dynamics. In r-LPC, the Koopman operator is used to form a global linear representation of the unknown dynamics, and an incremental actor-critic algorithm is presented for receding horizon optimization. To realize the satisfaction of system constraints, soft logarithmic barrier functions are designed within the learning predictive framework. The recursive feasibility and stability of the closed-loop system are discussed under the convergence arguments of the approximation algorithms adopted. Also, the robustness property of r-LPC is analyzed theoretically by taking into consideration the existence of perturbations on the controller due to possible approximation errors. Simulation results with the proposed learning control approach for the data-driven regulation of a Van der Pol oscillator system have been reported, including the comparisons with a classic MPC and an infinite-horizon Dual Heuristic Programming (DHP) algorithm. The results show that the r-LPC significantly outperforms the DHP algorithm in terms of control performance and can be comparative to the MPC in terms of regulating control as well as energy consumption. Moreover, its average computational cost is much smaller than that with the MPC in the adopted environment.

[1]  Frank Allgöwer,et al.  Learning an Approximate Model Predictive Controller With Guarantees , 2018, IEEE Control Systems Letters.

[2]  Haibo He,et al.  Functional Nonlinear Model Predictive Control Based on Adaptive Dynamic Programming , 2019, IEEE Transactions on Cybernetics.

[3]  Chuanqiang Lian,et al.  Learning-Based Predictive Control for Discrete-Time Nonlinear Systems With Stochastic Disturbances , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[4]  Lorenzo Fagiano,et al.  Learning-based predictive control for linear systems: a unitary approach , 2018, Autom..

[5]  Johan Löfberg,et al.  YALMIP : a toolbox for modeling and optimization in MATLAB , 2004 .

[6]  David Q. Mayne,et al.  Constrained model predictive control: Stability and optimality , 2000, Autom..

[7]  Shaocheng Tong,et al.  Neural Network Control-Based Adaptive Learning Design for Nonlinear Systems With Full-State Constraints , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[8]  Hao Xu,et al.  Finite-Horizon Near-Optimal Output Feedback Neural Network Control of Quantized Nonlinear Discrete-Time Systems With Input Constraint , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[9]  Haibo He,et al.  GrDHP: A General Utility Function Representation for Dual Heuristic Dynamic Programming , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[10]  Francesco Borrelli,et al.  Learning Model Predictive Control for Iterative Tasks. A Data-Driven Control Framework , 2016, IEEE Transactions on Automatic Control.

[11]  Shaocheng Tong,et al.  Adaptive control-based Barrier Lyapunov Functions for a class of stochastic nonlinear systems with full state constraints , 2018, Autom..

[12]  Hao Xu,et al.  Neural Network-Based Finite-Horizon Optimal Control of Uncertain Affine Nonlinear Discrete-Time Systems , 2015, IEEE Trans. Neural Networks Learn. Syst..

[13]  David Q. Mayne,et al.  Robust model predictive control of constrained linear systems with bounded disturbances , 2005, Autom..

[14]  Morten Hovd,et al.  On feasible sets for MPC and their approximations , 2011, Autom..

[15]  Christian Feller,et al.  Relaxed Logarithmic Barrier Function Based Model Predictive Control of Linear Systems , 2015, IEEE Transactions on Automatic Control.

[16]  C. L. Philip Chen,et al.  Adaptive Neural Control of Uncertain MIMO Nonlinear Systems With State and Input Constraints , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[17]  Torsten Koller,et al.  Learning-based Model Predictive Control for Safe Exploration and Reinforcement Learning , 2019, ArXiv.

[18]  Haibo He,et al.  Online Learning Control Using Adaptive Critic Designs With Sparse Kernel Machines , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[19]  Louis Wehenkel,et al.  Reinforcement Learning Versus Model Predictive Control: A Comparison on a Power System Problem , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[20]  Huaguang Zhang,et al.  Neural-Network-Based Near-Optimal Control for a Class of Discrete-Time Affine Nonlinear Systems With Control Constraints , 2009, IEEE Transactions on Neural Networks.

[21]  Shaocheng Tong,et al.  Reinforcement Learning Design-Based Adaptive Tracking Control With Less Learning Parameters for Nonlinear Discrete-Time MIMO Systems , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[22]  Huaguang Zhang,et al.  Adaptive Dynamic Programming: An Introduction , 2009, IEEE Computational Intelligence Magazine.

[23]  Huaguang Zhang,et al.  A Novel Infinite-Time Optimal Tracking Control Scheme for a Class of Discrete-Time Nonlinear Systems via the Greedy HDP Iteration Algorithm , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[24]  Ali Heydari,et al.  Finite-Horizon Control-Constrained Nonlinear Optimal Control Using Single Network Adaptive Critics , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[25]  Haibo He,et al.  Manifold-Based Reinforcement Learning via Locally Linear Reconstruction , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[26]  Igor Mezic,et al.  Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control , 2016, Autom..

[27]  Shalabh Bhatnagar,et al.  Natural actor-critic algorithms , 2009, Autom..

[28]  Clarence W. Rowley,et al.  A Data–Driven Approximation of the Koopman Operator: Extending Dynamic Mode Decomposition , 2014, Journal of Nonlinear Science.

[29]  P. Varaiya,et al.  Ellipsoidal Toolbox (ET) , 2006, Proceedings of the 45th IEEE Conference on Decision and Control.

[30]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[31]  Derong Liu,et al.  Policy Iteration Adaptive Dynamic Programming Algorithm for Discrete-Time Nonlinear Systems , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[32]  S. Joe Qin,et al.  A survey of industrial model predictive control technology , 2003 .

[33]  Adrian Wills,et al.  Barrier function based model predictive control , 2004, Autom..

[34]  Marcello Farina,et al.  A hierarchical multi-rate MPC scheme for interconnected systems , 2018, Autom..

[35]  Derong Liu,et al.  Neural-Network-Based Optimal Control for a Class of Unknown Discrete-Time Nonlinear Systems Using Globalized Dual Heuristic Programming , 2012, IEEE Transactions on Automation Science and Engineering.

[36]  Enoch Yeung,et al.  A Koopman Operator Approach for Computing and Balancing Gramians for Discrete Time Nonlinear Systems , 2017, 2018 Annual American Control Conference (ACC).

[37]  Haibo He,et al.  Adaptive Critic Nonlinear Robust Control: A Survey , 2017, IEEE Transactions on Cybernetics.

[38]  Lorenzo Fagiano,et al.  Scenario and Adaptive Model Predictive Control of Uncertain Systems , 2015 .

[39]  Zhong-Ping Jiang,et al.  Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics , 2012, Autom..

[40]  Igor Mezic,et al.  On Convergence of Extended Dynamic Mode Decomposition to the Koopman Operator , 2017, J. Nonlinear Sci..

[41]  Jan M. Maciejowski,et al.  Learning-based Nonlinear Model Predictive Control , 2017 .

[42]  Qinglai Wei,et al.  Optimal control of unknown nonaffine nonlinear discrete-time systems based on adaptive dynamic programming , 2012, Autom..

[43]  Vivek S. Borkar,et al.  The actor-critic algorithm as multi-time-scale stochastic approximation , 1997 .

[44]  Frank Allgöwer,et al.  Adaptive Model Predictive Control with Robust Constraint Satisfaction , 2017 .

[45]  Alberto Bemporad,et al.  Min-max control of constrained uncertain discrete-time linear systems , 2003, IEEE Trans. Autom. Control..

[46]  Bernhard Schölkopf,et al.  Statistical Learning Theory: Models, Concepts, and Results , 2008, Inductive Logic.

[47]  Philip S. Thomas,et al.  Learning Action Representations for Reinforcement Learning , 2019, ICML.