Reinforcement Learning Design-Based Adaptive Tracking Control With Less Learning Parameters for Nonlinear Discrete-Time MIMO Systems

Based on the neural network (NN) approximator, an online reinforcement learning algorithm is proposed for a class of affine multiple input and multiple output (MIMO) nonlinear discrete-time systems with unknown functions and disturbances. In the design procedure, two networks are provided where one is an action network to generate an optimal control signal and the other is a critic network to approximate the cost function. An optimal control signal and adaptation laws can be generated based on two NNs. In the previous approaches, the weights of critic and action networks are updated based on the gradient descent rule and the estimations of optimal weight vectors are directly adjusted in the design. Consequently, compared with the existing results, the main contributions of this paper are: (1) only two parameters are needed to be adjusted, and thus the number of the adaptation laws is smaller than the previous results and (2) the updating parameters do not depend on the number of the subsystems for MIMO systems and the tuning rules are replaced by adjusting the norms on optimal weight vectors in both action and critic networks. It is proven that the tracking errors, the adaptation laws, and the control inputs are uniformly bounded using Lyapunov analysis method. The simulation examples are employed to illustrate the effectiveness of the proposed algorithm.

[1]  Thomas J. Walsh,et al.  Exploring compact reinforcement-learning representations with linear regression , 2009, UAI.

[2]  Derong Liu,et al.  Online Synchronous Approximate Optimal Learning Algorithm for Multi-Player Non-Zero-Sum Games With Unknown Dynamics , 2014, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[3]  Yongduan Song,et al.  Cooperative Tracking Control of Nonlinear Multiagent Systems Using Self-Structuring Neural Networks , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[4]  Shaocheng Tong,et al.  Adaptive Neural Output Feedback Tracking Control for a Class of Uncertain Discrete-Time Nonlinear Systems , 2011, IEEE Transactions on Neural Networks.

[5]  Frank L. Lewis,et al.  Multilayer neural-net robot controller with guaranteed tracking performance , 1996, IEEE Trans. Neural Networks.

[6]  S. S. Ge,et al.  Synchronised tracking control of multi-agent system with high order dynamics , 2012 .

[7]  Bin Zhou,et al.  Stabilization of Discrete-Time Systems With Multiple Actuator Delays and Saturations , 2013, IEEE Transactions on Circuits and Systems I: Regular Papers.

[8]  Huaguang Zhang,et al.  Optimal Tracking Control for a Class of Nonlinear Discrete-Time Systems With Time Delays Based on Heuristic Dynamic Programming , 2011, IEEE Transactions on Neural Networks.

[9]  Junfei Qiao,et al.  A Self-Organizing Fuzzy Neural Network Based on a Growing-and-Pruning Algorithm , 2010, IEEE Transactions on Fuzzy Systems.

[10]  Shuzhi Sam Ge,et al.  Boundary control of a flexible marine installation system , 2011, 49th IEEE Conference on Decision and Control (CDC).

[11]  Xin Xu,et al.  Kernel-Based Least Squares Policy Iteration for Reinforcement Learning , 2007, IEEE Transactions on Neural Networks.

[12]  C. L. Philip Chen,et al.  An integration of neural network and rule-based systems for design and planning of mechanical assemblies , 1993, IEEE Trans. Syst. Man Cybern..

[13]  Bin Zhou,et al.  Discrete-time l∞ and l2 norm vanishment and low gain feedback with their applications in constrained control , 2012, 2012 24th Chinese Control and Decision Conference (CCDC).

[14]  Zongli Lin,et al.  Discrete-time l ∞ and l 2 norm vanishment and low gain feedback with their applications in constrained control , 2012, CCDC 2012.

[15]  Huaguang Zhang,et al.  A Novel Infinite-Time Optimal Tracking Control Scheme for a Class of Discrete-Time Nonlinear Systems via the Greedy HDP Iteration Algorithm , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[16]  Yasar Becerikli,et al.  Trajectory priming with dynamic fuzzy networks in nonlinear optimal control , 2004, IEEE Transactions on Neural Networks.

[17]  Jagannathan Sarangapani,et al.  Neural Network Control of Nonlinear Discrete-Time Systems , 2018 .

[18]  Youxian Sun,et al.  Universal Neural Network Control of MIMO Uncertain Nonlinear Systems , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[19]  Derong Liu,et al.  Neural-network-observer-based optimal control for unknown nonlinear systems using adaptive dynamic programming , 2013, Int. J. Control.

[20]  Derong Liu,et al.  Adaptive Critic Learning Techniques for Engine Torque and Air–Fuel Ratio Control , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[21]  Licheng Jiao,et al.  Adaptive Tracking for Periodically Time-Varying and Nonlinearly Parameterized Systems Using Multilayer Neural Networks , 2010, IEEE Transactions on Neural Networks.

[22]  Lei Yang,et al.  Direct Heuristic Dynamic Programming for Nonlinear Tracking Control With Filtered Tracking Error , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[23]  Guo-Xing Wen,et al.  Fuzzy Neural Network-Based Adaptive Control for a Class of Uncertain Nonlinear Stochastic Systems , 2014, IEEE Transactions on Cybernetics.

[24]  Frank L. Lewis,et al.  Neural net robot controller with guaranteed tracking performance , 1995, IEEE Trans. Neural Networks.

[25]  Frank L. Lewis,et al.  Model-free Q-learning designs for linear discrete-time zero-sum games with application to H-infinity control , 2007, Autom..

[26]  Junsheng Ren,et al.  Adaptive fuzzy robust tracking controller design via small gain approach and its application , 2003, IEEE Trans. Fuzzy Syst..

[27]  Shuzhi Sam Ge,et al.  Robust Adaptive Neural Network Control for a Class of Uncertain MIMO Nonlinear Systems With Input Nonlinearities , 2010, IEEE Transactions on Neural Networks.

[28]  Shuzhi Sam Ge,et al.  Output Feedback NN Control for Two Classes of Discrete-Time Systems With Unknown Control Directions in a Unified Approach , 2008, IEEE Transactions on Neural Networks.

[29]  Zhongke Shi,et al.  Reinforcement Learning Output Feedback NN Control Using Deterministic Learning Technique , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[30]  Qinmin Yang,et al.  Reinforcement Learning Controller Design for Affine Nonlinear Discrete-Time Systems using Online Approximators , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[31]  Li-Xin Wang.  Stable adaptive fuzzy control of nonlinear systems , 1993, IEEE Trans. Fuzzy Syst..

[32]  Xuemei Ren,et al.  Identification of Extended Hammerstein Systems Using Dynamic Self-Optimizing Neural Networks , 2011, IEEE Transactions on Neural Networks.

[33]  Li-Xin Wang,et al.  Adaptive fuzzy systems and control - design and stability analysis , 1994 .

[34]  Derong Liu,et al.  Neural-Network-Based Optimal Control for a Class of Unknown Discrete-Time Nonlinear Systems Using Globalized Dual Heuristic Programming , 2012, IEEE Transactions on Automation Science and Engineering.

[35]  Derong Liu,et al.  Finite-Approximation-Error-Based Optimal Control Approach for Discrete-Time Nonlinear Systems , 2013, IEEE Transactions on Cybernetics.

[36]  Ye Zhao,et al.  Asynchronous Filtering of Discrete-Time Switched Linear Systems With Average Dwell Time , 2011, IEEE Transactions on Circuits and Systems I: Regular Papers.

[37]  Tao Zhang,et al.  Stable Adaptive Neural Network Control , 2001, The Springer International Series on Asian Studies in Computer and Information Science.

[38]  Frank L. Lewis,et al.  Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach , 2005, Autom..

[39]  Shuzhi Sam Ge,et al.  Robust Adaptive Boundary Control of a Vibrating String Under Unknown Time-Varying Disturbance , 2010, IEEE Transactions on Control Systems Technology.

[40]  Shuzhi Sam Ge,et al.  Adaptive boundary control of a flexible marine installation system , 2010, Autom..

[41]  Jooyoung Park,et al.  Universal Approximation Using Radial-Basis-Function Networks , 1991, Neural Computation.

[42]  Chun-Yi Su,et al.  Neural-Adaptive Control of Single-Master–Multiple-Slaves Teleoperation for Coordinated Multiple Mobile Manipulators With Time-Varying Communication Delays and Input Uncertainties , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[43]  F. Lewis,et al.  Discrete-time neural net controller for a class of nonlinear dynamical systems , 1996, IEEE Trans. Autom. Control..

[44]  Derong Liu,et al.  Decentralized Stabilization for a Class of Continuous-Time Nonlinear Interconnected Systems Using Online Learning Optimal Control Approach , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[45]  Shaocheng Tong,et al.  Adaptive Neural Output Feedback Controller Design With Reduced-Order Observer for a Class of Uncertain Nonlinear SISO Systems , 2011, IEEE Transactions on Neural Networks.

[46]  Shuzhi Sam Ge,et al.  Adaptive Predictive Control Using Neural Network for a Class of Pure-Feedback Systems in Discrete Time , 2008, IEEE Transactions on Neural Networks.

[47]  Derong Liu,et al.  Numerical adaptive learning control scheme for discrete-time non-linear systems , 2013 .

[48]  Shuzhi Sam Ge,et al.  Adaptive NN control of uncertain nonlinear pure-feedback systems , 2002, Autom..

[49]  Frank L. Lewis,et al.  Neural network compensation control for mechanical systems with disturbances , 2009, Autom..

[50]  Jean-Jacques E. Slotine,et al.  Neural Network Control of Unknown Nonlinear Systems , 1989, 1989 American Control Conference.

[51]  Shengyuan Xu,et al.  Neural-Network-Based Decentralized Adaptive Output-Feedback Control for Large-Scale Stochastic Nonlinear Systems , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[52]  Shengyuan Xu,et al.  Adaptive Output-Feedback Fuzzy Tracking Control for a Class of Nonlinear Systems , 2011, IEEE Transactions on Fuzzy Systems.

[53]  Jennie Si,et al.  Online learning control by association and reinforcement. , 2001, IEEE transactions on neural networks.

[54]  Shuzhi Sam Ge,et al.  Adaptive Neural Control for a Class of Nonlinear Systems With Uncertain Hysteresis Inputs and Time-Varying State Delays , 2009, IEEE Transactions on Neural Networks.

[55]  Frank L. Lewis,et al.  Adaptive optimal control for continuous-time linear systems based on policy iteration , 2009, Autom..

[56]  Zhijun Li,et al.  Adaptive fuzzy control for synchronization of nonlinear teleoperators with stochastic time-varying communication delays , 2011, 2011 IEEE International Conference on Robotics and Automation.

[57]  C. L. Philip Chen,et al.  An incremental adaptive implementation of functional-link processing for function approximation, time-series prediction, and system identification , 1998, Neurocomputing.

[58]  Frank L. Lewis,et al.  2009 Special Issue: Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinear systems , 2009 .

[59]  D. Liu,et al.  Adaptive Dynamic Programming for Finite-Horizon Optimal Control of Discrete-Time Nonlinear Systems With $\varepsilon$-Error Bound , 2011, IEEE Transactions on Neural Networks.

[60]  Lei Yang,et al.  Performance Evaluation of Direct Heuristic Dynamic Programming using Control-Theoretic Measures , 2009, J. Intell. Robotic Syst..

[61]  Junmin Li,et al.  Decentralized Output-Feedback Neural Control for Systems With Unknown Interconnections , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[62]  Haibo He,et al.  Online Learning Control Using Adaptive Critic Designs With Sparse Kernel Machines , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[63]  Guo-Xing Wen,et al.  Adaptive Consensus Control for a Class of Nonlinear Multiagent Time-Delay Systems Using Neural Networks , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[64]  Huaguang Zhang,et al.  Neural-Network-Based Near-Optimal Control for a Class of Discrete-Time Affine Nonlinear Systems With Control Constraints , 2009, IEEE Transactions on Neural Networks.

[65]  Derong Liu,et al.  Policy Iteration Adaptive Dynamic Programming Algorithm for Discrete-Time Nonlinear Systems , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[66]  Huaguang Zhang,et al.  A fuzzy basis function vector-based multivariable adaptive controller for nonlinear systems , 2000, IEEE Trans. Syst. Man Cybern. Part B.