Neural Networks in Feedback Control Systems

This chapter shows how neural networks (NNs) fulfill the promise of providing model-free learning controllers for a class of nonlinear systems in the sense that a structural or parameterized model of the system dynamics is not needed. It shows different methods of NN controller design that yield guaranteed performance for systems of different structure and complexity. The control structures discussed in the chapter are multiloop controllers with NNs in some of the loops and an outer tracking unity-gain feedback loop. Throughout, there are repeatable design algorithms and guarantees of system performance including both small tracking errors and bounded NN weights. Recent results show that approximate dynamic programming (ADP) with critic and actor neural networks, allows the design of adaptive learning controllers that converge online to optimal control solutions, while also guaranteeing closed-loop stability. ADP also offers design methods for differential games that can be implemented online in real time. Keywords: adaptive learning controllers; approximate dynamic programming (ADP); dynamic games; feedback control systems; neural network feedback control structures; neural networks (NNs); nonlinear systems

[1]  George A. Rovithakis,et al.  Performance of a neural adaptive tracking controller for multi-input nonlinear dynamical systems in the presence of additive and multiplicative external disturbances , 2000, IEEE Trans. Syst. Man Cybern. Part A.

[2]  Frank L. Lewis,et al.  ROBUST ADAPTIVE CONTROL OF ROBOTS USING NEURAL NETWORK: GLOBAL STABILITY , 2008 .

[3]  J. Tsitsiklis,et al.  Efficient algorithms for globally optimal trajectories , 1994, Proceedings of 1994 33rd IEEE Conference on Decision and Control.

[4]  Randal W. Beard,et al.  Galerkin approximations of the generalized Hamilton-Jacobi-Bellman equation , 1997, Autom..

[5]  Héctor J. Sussmann,et al.  Uniqueness of the weights for minimal feedforward nets with a given input-output map , 1992, Neural Networks.

[6]  Frank L. Lewis,et al.  Implementation of a neural network tracking controller for a single flexible link: comparison with PD and PID controllers , 1998, IEEE Trans. Ind. Electron..

[7]  Kumpati S. Narendra,et al.  Adaptive control using multiple models , 1997, IEEE Trans. Autom. Control..

[8]  Shuzhi Sam Ge,et al.  An ISS-modular approach for adaptive neural control of pure-feedback systems , 2006, Autom..

[9]  Ronald A. Howard,et al.  Dynamic Programming and Markov Processes , 1960 .

[10]  S.H.G. ten Hagen,et al.  Linear Quadratic Regulation using reinforcement learning , 1998 .

[11]  R. Sanner,et al.  Structurally dynamic wavelet networks for adaptive control of robotic systems , 1998 .

[12]  J. W. Modestino,et al.  Adaptive Control , 1998 .

[13]  Sergey Edward Lyshevski.  Control Systems Theory with Engineering Applications , 2001 .

[14]  Nader Sadegh,et al.  A perceptron network for functional identification and control of nonlinear systems , 1993, IEEE Trans. Neural Networks.

[15]  J. Slotine,et al.  On the Adaptive Control of Robot Manipulators , 1987 .

[16]  A. Schaft.  L/sub 2/-gain analysis of nonlinear systems and nonlinear state-feedback H/sub infinity / control , 1992 .

[17]  Richard S. Sutton,et al.  Computational Schemes and Neural Network Models for Formation and Control of Multijoint Arm Trajectory , 1995 .

[18]  Anthony J. Calise,et al.  Adaptive nonlinear control of agile antiair missiles using neural networks , 2000, IEEE Trans. Control. Syst. Technol..

[19]  Frank L. Lewis,et al.  Feedback linearization using neural networks , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[20]  Kenneth J. Hunt,et al.  Neural Adaptive Control Technology , 1996 .

[21]  Jennie Si,et al.  Online learning control by association and reinforcement. , 2001, IEEE transactions on neural networks.

[22]  C. R. Johnson,et al.  Lectures on adaptive parameter estimation , 1988 .

[23]  Frank L. Lewis,et al.  Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem , 2010, Autom..

[24]  Thomas Parisini,et al.  Nonlinear stabilization by receding-horizon neural regulators , 1995, Proceedings of 1995 34th IEEE Conference on Decision and Control.

[25]  Richard S. Sutton,et al.  Learning to predict by the methods of temporal differences , 1988, Machine Learning.

[26]  Jean-Jacques E. Slotine,et al.  Adaptive sliding controller synthesis for non-linear systems , 1986 .

[27]  Kenji Doya,et al.  Neural mechanisms of learning and control , 2001 .

[28]  Frank L. Lewis,et al.  Aircraft Control and Simulation , 1992 .

[29]  L. Hunt,et al.  Global transformations of nonlinear systems , 1983 .

[30]  Paul J. Werbos,et al.  Neural networks for control and system identification , 1989, Proceedings of the 28th IEEE Conference on Decision and Control,.

[31]  Tomas Landelius,et al.  Reinforcement Learning and Distributed Local Model Synthesis , 1997 .

[32]  J F Blackburn,et al.  Robot Manipulator Control. , 1983 .

[33]  Eduardo D. Sontag,et al.  Neural Networks for Control , 1993 .

[34]  Richard S. Sutton,et al.  Connectionist Learning for Control , 1995 .

[35]  Robert F. Stengel,et al.  An adaptive critic global controller , 2002, Proceedings of the 2002 American Control Conference (IEEE Cat. No.CH37301).

[36]  Richard D. Braatz,et al.  On the "Identification and control of dynamical systems using neural networks" , 1997, IEEE Trans. Neural Networks.

[37]  N. Harris McClamroch,et al.  Feedback Stabilization and Tracking of Constrained Robots , 1987, 1987 American Control Conference.

[38]  Frank L. Lewis,et al.  Nearly optimal state feedback control of constrained nonlinear systems using a neural networks HJB approach , 2004, Annu. Rev. Control..

[39]  Anthony J. Calise,et al.  Adaptive output feedback control of nonlinear systems using neural networks , 2001, Autom..

[40]  L. C. Baird,et al.  Reinforcement learning in continuous time: advantage updating , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[41]  Alexander S. Poznyak,et al.  Nonlinear adaptive trajectory tracking using dynamic neural networks , 1999, IEEE Trans. Neural Networks.

[42]  Li-Xin Wang,et al.  Adaptive fuzzy systems and control - design and stability analysis , 1994 .

[43]  Mitsuo Kawato,et al.  Computational schemes and neural network models for formulation and control of multijoint arm trajectory , 1990 .

[44]  Frank L. Lewis,et al.  Policy Iterations on the Hamilton–Jacobi–Isaacs Equation for $H_{\infty}$ State Feedback Control With Input Saturation , 2006, IEEE Transactions on Automatic Control.

[45]  Mitsuo Kawato,et al.  Feedback-error-learning neural network for trajectory control of a robotic manipulator , 1988, Neural Networks.

[46]  Frank L. Lewis,et al.  Neurodynamic Programming and Zero-Sum Games for Constrained Control Systems , 2008, IEEE Transactions on Neural Networks.

[47]  Douglas C. Hittle,et al.  Robust reinforcement learning control with static and dynamic stability , 2001 .

[48]  Andrew G. Barto,et al.  Adaptive linear quadratic control using policy iteration , 1994, Proceedings of 1994 American Control Conference - ACC '94.

[49]  George G. Lendaris,et al.  Adaptive dynamic programming , 2002, IEEE Trans. Syst. Man Cybern. Part C.

[50]  Frank L. Lewis,et al.  Neuro-Fuzzy Control of Industrial Systems with Actuator Nonlinearities , 1987 .

[51]  Manolis A. Christodoulou,et al.  Adaptive control of unknown plants using dynamical neural networks , 1994, IEEE Trans. Syst. Man Cybern..

[52]  Shuzhi Sam Ge,et al.  Adaptive NN control for a class of strict-feedback discrete-time nonlinear systems , 2003, Autom..

[53]  Dan Wang,et al.  Adaptive neural network control for a class of uncertain nonlinear systems in pure-feedback form , 2002, Autom..

[54]  I. Kanellakopoulos,et al.  Systematic Design of Adaptive Controllers for Feedback Linearizable Systems , 1991, 1991 American Control Conference.

[55]  Richard S. Sutton,et al.  A Menu of Designs for Reinforcement Learning Over Time , 1995 .

[56]  Marios M. Polycarpou,et al.  Stable adaptive neural control scheme for nonlinear systems , 1996, IEEE Trans. Autom. Control..

[57]  Shuzhi Sam Ge,et al.  Output Feedback NN Control for Two Classes of Discrete-Time Systems With Unknown Control Directions in a Unified Approach , 2008, IEEE Transactions on Neural Networks.

[58]  F. Lewis,et al.  Neural Network Control of Robot Arms and Nonlinear Systems , 1997 .

[59]  Petros A. Ioannou,et al.  Neural networks as on-line approximators of nonlinear systems , 1992, [1992] Proceedings of the 31st IEEE Conference on Decision and Control.

[60]  Shuzhi Sam Ge,et al.  Adaptive NN control of uncertain nonlinear pure-feedback systems , 2002, Autom..

[61]  Andrew G. Barto,et al.  Connectionist learning for control , 1990 .

[62]  K. Narendra,et al.  A New Adaptive Law for Robust Adaptation without Persistent Excitation , 1986, 1986 American Control Conference.

[63]  Robert M. Sanner,et al.  Gaussian Networks for Direct Adaptive Control , 1991, 1991 American Control Conference.

[64]  Bin Yao,et al.  Neural network adaptive robust control of nonlinear systems in semi-strict feedback form , 2001, Proceedings of the 2001 American Control Conference. (Cat. No.01CH37148).

[65]  Frank L. Lewis,et al.  Neural Network Control Of Robot Manipulators And Non-Linear Systems , 1998 .

[66]  Warren B. Powell,et al.  Reinforcement Learning and Its Relationship to Supervised Learning , 2004 .

[67]  Frank L. Lewis,et al.  High-Level Feedback Control with Neural Networks , 1998, World Scientific Series in Robotics and Intelligent Systems.

[68]  F.L. Lewis,et al.  Implementation of a neural net tracking controller for a single flexible link: comparison with PD and PID controllers , 1997, Proceedings of the 1997 American Control Conference (Cat. No.97CH36041).

[69]  F. L. Lewis.  NONLINEAR NETWORK STRUCTURES FOR FEEDBACK CONTROL , 1999 .

[70]  Jay A. Farrell,et al.  Nonlinear adaptive control using networks of piecewise linear approximators , 2000, IEEE Trans. Neural Networks Learn. Syst..

[71]  Keng Peng Tee,et al.  Adaptive Neural Control for Output Feedback Nonlinear Systems Using a Barrier Lyapunov Function , 2010, IEEE Transactions on Neural Networks.

[72]  Radhakant Padhi,et al.  Adaptive-critic based optimal neuro control synthesis for distributed parameter systems , 2001, Autom..

[73]  Warren B. Powell,et al.  Handbook of Learning and Approximate Dynamic Programming , 2006, IEEE Transactions on Automatic Control.

[74]  Frank L. Lewis,et al.  Adaptive optimal control for continuous-time linear systems based on policy iteration , 2009, Autom..

[75]  Tao Zhang,et al.  Stable Adaptive Neural Network Control , 2001, The Springer International Series on Asian Studies in Computer and Information Science.

[76]  Thomas Parisini,et al.  Stable hybrid control based on discrete-event automata and receding-horizon neural regulators , 2001, Autom..

[77]  Jooyoung Park,et al.  Universal Approximation Using Radial-Basis-Function Networks , 1991, Neural Computation.

[78]  Jay A. Farrell,et al.  Stability and approximator convergence in nonparametric nonlinear adaptive control , 1998, IEEE Trans. Neural Networks.

[79]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[80]  Jie Huang,et al.  Neural network enhanced output regulation in nonlinear systems , 2001, Autom..

[81]  Shuzhi Sam Ge,et al.  Adaptive neural network control for strict-feedback nonlinear systems using backstepping design , 1999, Proceedings of the 1999 American Control Conference (Cat. No. 99CH36251).

[82]  Anthony J. Calise,et al.  Analysis of Adaptive Neural Networks for Helicopter Flight Control , 1997 .

[83]  Shuzhi Sam Ge,et al.  Adaptive MNN control for a class of non-affine NARMAX systems with disturbances , 2004, Syst. Control. Lett..

[84]  D. Kleinman.  On an iterative technique for Riccati equation computations , 1968 .

[85]  K. Hunt.  APPLIED OPTIMAL CONTROL AND ESTIMATION : DIGITAL DESIGN AND IMPLEMENTATION , 1993 .

[86]  George G. Lendaris,et al.  Globally convergent approximate dynamic programming applied to an autolander , 2001, Proceedings of the 2001 American Control Conference. (Cat. No.01CH37148).

[87]  John N. Tsitsiklis,et al.  Neuro-Dynamic Programming , 1996, Encyclopedia of Machine Learning.

[88]  Yoh-Han Pao,et al.  Stochastic choice of basis functions in adaptive function approximation and the functional-link net , 1995, IEEE Trans. Neural Networks.

[89]  Kumpati S. Narendra,et al.  Gradient methods for the optimization of dynamical systems containing neural networks , 1991, IEEE Trans. Neural Networks.

[90]  Jianming Lu,et al.  Adaptive Control for Nonlinear Systems Using Neural Networks , 1999 .

[91]  Draguna Vrabie,et al.  Adaptive optimal controllers based on Generalized Policy Iteration in a continuous-time framework , 2009, 2009 17th Mediterranean Conference on Control and Automation.

[92]  Kenji Doya,et al.  Reinforcement Learning in Continuous Time and Space , 2000, Neural Computation.

[93]  Frank L. Lewis,et al.  Multilayer neural-net robot controller with guaranteed tracking performance , 1996, IEEE Trans. Neural Networks.

[94]  Shuzhi Sam Ge,et al.  Adaptive Predictive Control Using Neural Network for a Class of Pure-Feedback Systems in Discrete Time , 2008, IEEE Transactions on Neural Networks.

[95]  Shuzhi Sam Ge,et al.  Adaptive Neural Network Control For Smart Materials Robots Using Singular Perturbation Technique , 2008 .

[96]  Shuzhi Sam Ge,et al.  Adaptive neural control of uncertain MIMO nonlinear systems , 2004, IEEE Transactions on Neural Networks.

[97]  Shuzhi Sam Ge,et al.  Adaptive tracking control of uncertain MIMO nonlinear systems with input constraints , 2011, Autom..

[98]  Jerry M. Mendel,et al.  Reinforcement-learning control and pattern recognition systems , 1994 .

[99]  Frank L. Lewis,et al.  Applied Optimal Control and Estimation , 1992 .

[100]  Shuzhi Sam Ge,et al.  Adaptive Neural Control for a Class of Nonlinear Systems With Uncertain Hysteresis Inputs and Time-Varying State Delays , 2009, IEEE Transactions on Neural Networks.

[101]  Frank L. Lewis,et al.  Multilayer discrete-time neural-net controller with guaranteed performance , 1996, IEEE Trans. Neural Networks.

[102]  Frank L. Lewis,et al.  Adaptive critic neural network for feedforward compensation , 1999, Proceedings of the 1999 American Control Conference (Cat. No. 99CH36251).

[103]  Frank L. Lewis,et al.  Introduction to the special issue on neural network feedback control , 2001, Autom..

[104]  George A. Rovithakis.  Stable adaptive neuro-control design via Lyapunov function derivative estimation , 2001, Autom..

[105]  Ben J. A. Kröse,et al.  Learning from delayed rewards , 1995, Robotics Auton. Syst..

[106]  Narasimhan Sundararajan,et al.  Neuro-controller design for nonlinear fighter aircraft maneuver using fully tuned RBF networks , 2001, Autom..

[107]  Tamer Basar,et al.  Disturbance attenuating controller design for strict-feedback systems with structurally unknown dynamics , 2001, Autom..

[108]  Frank L. Lewis,et al.  Multi-player non-zero-sum games: Online adaptive learning solution of coupled Hamilton-Jacobi equations , 2011, Autom..

[109]  Jun Wang,et al.  Recurrent neural networks for nonlinear output regulation , 2001, Autom..

[110]  Frank L. Lewis,et al.  Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach , 2005, Autom..

[111]  P. Kokotovic.  Applications of Singular Perturbation Techniques to Control Problems , 1984 .

[112]  Jin Zhang,et al.  Neural-network control of nonaffine nonlinear system with zero dynamics by state and output feedback , 2003, IEEE Trans. Neural Networks.

[113]  Peter Dayan,et al.  The convergence of TD(λ) for general λ , 1992, Machine Learning.

[114]  Jie Huang,et al.  Neural network based adaptive tracking of uncertain nonlinear systems in triangular form , 2001, Proceedings of the 2001 American Control Conference. (Cat. No.01CH37148).

[115]  James S. Albus,et al.  New Approach to Manipulator Control: The Cerebellar Model Articulation Controller (CMAC)1 , 1975 .

[116]  Lotfi A. Zadeh,et al.  General System Theory , 1962 .

[117]  Duc Truong Pham,et al.  Adaptive control of dynamic systems using neural networks , 1993, Proceedings of IEEE Systems Man and Cybernetics Conference - SMC.

[118]  Frank L. Lewis,et al.  Neural net robot controller with guaranteed tracking performance , 1995, IEEE Trans. Neural Networks.

[119]  Frank L. Lewis,et al.  Guest Editorial: Special Issue on Adaptive Dynamic Programming and Reinforcement Learning in Feedback Control , 2008, IEEE Trans. Syst. Man Cybern. Part B.

[120]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[121]  Andrew R. Barron,et al.  Universal approximation bounds for superpositions of a sigmoidal function , 1993, IEEE Trans. Inf. Theory.

[122]  Brian D. O. Anderson,et al.  A game theoretic algorithm to compute local stabilizing solutions to HJBI equations in nonlinear H∞ control , 2009, Autom..

[123]  Fu-Chuang Chen,et al.  Practical stability issues in CMAC neural network control systems , 1996, IEEE Trans. Control. Syst. Technol..

[124]  Hisham Abou-Kandil,et al.  On global existence of solutions to coupled matrix Riccati equations in closed-loop Nash games , 1996, IEEE Trans. Autom. Control..

[125]  Eduardo Sontag,et al.  For neural networks, function determines form , 1992, [1992] Proceedings of the 31st IEEE Conference on Decision and Control.

[126]  Lyle Noakes,et al.  Continuous-Time Adaptive Critics , 2007, IEEE Transactions on Neural Networks.

[127]  Tao Zhang,et al.  Adaptive neural network control of nonlinear systems by state and output feedback , 1999, IEEE Trans. Syst. Man Cybern. Part B.

[128]  Shuzhi Sam Ge,et al.  Adaptive Neural Network Control of Robotic Manipulators , 1999, World Scientific Series in Robotics and Intelligent Systems.

[129]  Paul J. Werbos,et al.  Approximate dynamic programming for real-time control and neural modeling , 1992 .

[130]  Lennart Ljung,et al.  On-line identification and adaptive trajectory tracking for nonlinear stochastic continuous time systems using differential neural networks , 2001, Autom..

[131]  George N. Saridis,et al.  An Approximation Theory of Optimal Control for Trainable Manipulators , 1979, IEEE Transactions on Systems, Man, and Cybernetics.

[132]  Danil V. Prokhorov,et al.  Analyzing for Lyapunov stability with adaptive critics , 1998, SMC'98 Conference Proceedings. 1998 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.98CH36218).

[133]  G. Saridis,et al.  Approximate Solutions to the Time-Invariant Hamilton–Jacobi–Bellman Equation , 1998 .