Adaptive Dynamic Programming and Adaptive Optimal Output Regulation of Linear Systems

This note studies the adaptive optimal output regulation problem for continuous-time linear systems, which aims to achieve asymptotic tracking and disturbance rejection by minimizing some predefined costs. Reinforcement learning and adaptive dynamic programming techniques are employed to compute an approximated optimal controller using input/partial-state data despite unknown system dynamics and unmeasurable disturbance. Rigorous stability analysis shows that the proposed controller exponentially stabilizes the closed-loop system and the output of the plant asymptotically tracks the given reference signal. Simulation results on a LCL coupled inverter-based distributed generation system demonstrate the effectiveness of the proposed approach.

[1]  Zhong-Ping Jiang,et al.  Adaptive dynamic programming and optimal control of nonlinear nonaffine systems , 2014, Autom..

[2]  Yu Jiang,et al.  Robust Adaptive Dynamic Programming and Feedback Stabilization of Nonlinear Systems , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[3]  Ioan Doré Landau,et al.  Indirect adaptive regulation strategy for the attenuation of time varying narrow-band disturbances applied to a benchmark problem , 2013, Eur. J. Control.

[4]  Xin Zhang,et al.  Data-Driven Robust Approximate Optimal Tracking Control for Unknown General Nonlinear Systems Using Adaptive Dynamic Programming Method , 2011, IEEE Transactions on Neural Networks.

[5]  Warren E. Dixon,et al.  Model-based reinforcement learning for approximate optimal regulation , 2016, Autom..

[6]  Huaguang Zhang,et al.  A Novel Infinite-Time Optimal Tracking Control Scheme for a Class of Discrete-Time Nonlinear Systems via the Greedy HDP Iteration Algorithm , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[7]  Warren B. Powell,et al.  Approximate Dynamic Programming: Solving the Curses of Dimensionality (Wiley Series in Probability and Statistics) , 2007 .

[8]  Zhong-Ping Jiang,et al.  Nonlinear Control of Dynamic Networks , 2014 .

[9]  F. Lewis,et al.  Reinforcement Learning and Feedback Control: Using Natural Decision Methods to Design Optimal Adaptive Controllers , 2012, IEEE Control Systems.

[10]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[11]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[12]  Stephen J. Finney,et al.  A Modified Stationary Reference Frame-Based Predictive Current Control With Zero Steady-State Error for LCL Coupled Inverter-Based Distributed Generation Systems , 2011, IEEE Transactions on Industrial Electronics.

[13]  Warren B. Powell,et al.  Approximate Dynamic Programming - Solving the Curses of Dimensionality , 2007 .

[14]  Alberto Isidori,et al.  Robust Autonomous Guidance: An Internal Model Approach , 2003 .

[15]  Zhengtao Ding.  Output regulation of uncertain nonlinear systems with nonlinear exosystems , 2006, IEEE Transactions on Automatic Control.

[16]  Zhong-Ping Jiang,et al.  A robust adaptive dynamic programming principle for sensorimotor control with signal-dependent noise , 2015, J. Syst. Sci. Complex..

[17]  Zhong-Ping Jiang,et al.  Global optimal output regulation of partially linear systems via robust adaptive dynamic programming , 2015 .

[18]  Lorenzo Marconi,et al.  Robust Autonomous Guidance , 2003 .

[19]  Frank L. Lewis,et al.  Optimal Control: Lewis/Optimal Control 3e , 2012 .

[20]  R. Zanasi,et al.  Output regulation of nonlinear systems by sliding mode , 2001, Autom..

[21]  Luigi Fortuna,et al.  Reinforcement Learning and Adaptive Dynamic Programming for Feedback Control , 2009 .

[22]  D. Mayne.  Nonlinear and Adaptive Control Design [Book Review] , 1996, IEEE Transactions on Automatic Control.

[23]  B. Francis.  The linear multivariable regulator problem , 1976, 1976 IEEE Conference on Decision and Control including the 15th Symposium on Adaptive Processes.

[24]  Huaguang Zhang,et al.  Online optimal tracking control of continuous-time linear systems with unknown dynamics by using adaptive dynamic programming , 2014, Int. J. Control.

[25]  Warren E. Dixon,et al.  Approximate optimal trajectory tracking for continuous-time nonlinear systems , 2013, Autom..

[26]  Zhong-Ping Jiang,et al.  Global Adaptive Dynamic Programming for Continuous-Time Nonlinear Systems , 2013, IEEE Transactions on Automatic Control.

[27]  Warren E. Dixon,et al.  Asymptotic tracking by a reinforcement learning-based adaptive critic controller , 2011 .

[28]  Warren B. Powell,et al.  “Approximate dynamic programming: Solving the curses of dimensionality” by Warren B. Powell , 2007, Wiley Series in Probability and Statistics.

[29]  D. Kleinman.  On an iterative technique for Riccati equation computations , 1968 .

[30]  Derong Liu,et al.  Neural-network-based optimal tracking control scheme for a class of unknown discrete-time nonlinear systems using iterative ADP algorithm , 2014, Neurocomputing.

[31]  Andrew G. Barto,et al.  Adaptive linear quadratic control using policy iteration , 1994, Proceedings of 1994 American Control Conference - ACC '94.

[32]  Zhong-Ping Jiang,et al.  Robust adaptive dynamic programming for linear and nonlinear systems: An overview , 2013, Eur. J. Control.

[33]  Jie Huang,et al.  Nonlinear Output Regulation: Theory and Applications , 2004 .

[34]  Sean P. Meyn,et al.  Q-learning and Pontryagin's Minimum Principle , 2009, Proceedings of the 48h IEEE Conference on Decision and Control (CDC) held jointly with 2009 28th Chinese Control Conference.

[35]  Zhong-Ping Jiang,et al.  Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics , 2012, Autom..

[36]  Arthur J. Krener.  The Construction of Optimal Linear and Nonlinear Regulators , 1992 .

[37]  Frank L. Lewis,et al.  Linear Quadratic Tracking Control of Partially-Unknown Continuous-Time Systems Using Reinforcement Learning , 2014, IEEE Transactions on Automatic Control.

[38]  Frank L. Lewis,et al.  Model-free Q-learning designs for linear discrete-time zero-sum games with application to H-infinity control , 2007, Autom..

[39]  Eduardo D. Sontag,et al.  Adaptation and regulation with signal detection implies internal model , 2003, Syst. Control. Lett..

[40]  Frank L. Lewis,et al.  Reinforcement Learning and Approximate Dynamic Programming for Feedback Control , 2012 .

[41]  Zhong-Ping Jiang,et al.  Adaptive dynamic programming as a theory of sensorimotor control , 2012, 2012 IEEE Signal Processing in Medicine and Biology Symposium (SPMB).

[42]  A. Saberi,et al.  On optimal output regulation for linear systems , 2003 .

[43]  Robert F. Stengel,et al.  An adaptive critic global controller , 2002, Proceedings of the 2002 American Control Conference (IEEE Cat. No.CH37301).