Event-Triggered Distributed Control of Nonlinear Interconnected Systems Using Online Reinforcement Learning With Exploration

In this paper, a distributed control scheme for an interconnected system composed of uncertain input affine nonlinear subsystems with event triggered state feedback is presented by using a novel hybrid learning scheme-based approximate dynamic programming with online exploration. First, an approximate solution to the Hamilton-Jacobi–Bellman equation is generated with event sampled neural network (NN) approximation and subsequently, a near optimal control policy for each subsystem is derived. Artificial NNs are utilized as function approximators to develop a suite of identifiers and learn the dynamics of each subsystem. The NN weight tuning rules for the identifier and event-triggering condition are derived using Lyapunov stability theory. Taking into account, the effects of NN approximation of system dynamics and boot-strapping, a novel NN weight update is presented to approximate the optimal value function. Finally, a novel strategy to incorporate exploration in online control framework, using identifiers, is introduced to reduce the overall cost at the expense of additional computations during the initial online learning phase. System states and the NN weight estimation errors are regulated and local uniformly ultimately bounded results are achieved. The analytical results are substantiated using simulation studies.

[1]  P. Dayan.  The Convergence of TD(λ) for General λ , 2004, Machine Learning.

[2]  P. J. Werbos.  Optimization methods for brain-like intelligent control , 1995, Proceedings of 1995 34th IEEE Conference on Decision and Control.

[3]  Sarangapani Jagannathan,et al.  Approximate optimal distributed control of uncertain nonlinear interconnected systems with event-sampled feedback , 2016, 2016 IEEE 55th Conference on Decision and Control (CDC).

[4]  Avimanyu Sahoo,et al.  Neural network-based adaptive event-triggered control of nonlinear continuous-time systems , 2013, 2013 IEEE International Symposium on Intelligent Control (ISIC).

[5]  Qing-Long Han,et al.  An Overview and Deep Investigation on Sampled-Data-Based Event-Triggered Control and Filtering for Networked Systems , 2017, IEEE Transactions on Industrial Informatics.

[6]  Qing-Long Han,et al.  Distributed event-triggered H1 filtering over sensor networks with communication delays , 2014 .

[7]  S. Sastry,et al.  Adaptive Control: Stability, Convergence and Robustness , 1989 .

[8]  Haibo He,et al.  Event-Triggered Adaptive Dynamic Programming for Continuous-Time Systems With Control Constraints , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[9]  Derong Liu,et al.  Neural-Network-Based Distributed Adaptive Robust Control for a Class of Nonlinear Multiagent Systems With Time Delays and External Noises , 2016, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[10]  Neural Network Control Of Robot Manipulators And Non Linear Systems Series In Systems And Control , 2016 .

[11]  Jennie Si,et al.  Online learning control by association and reinforcement. , 2001, IEEE transactions on neural networks.

[12]  Kenji Doya,et al.  Reinforcement Learning in Continuous Time and Space , 2000, Neural Computation.

[13]  Xiaofeng Wang,et al.  Event-Triggering in Distributed Networked Control Systems , 2011, IEEE Transactions on Automatic Control.

[14]  Avimanyu Sahoo,et al.  Approximate Optimal Control of Affine Nonlinear Continuous-Time Systems Using Event-Sampled Neurodynamic Programming , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[15]  Zhong-Ping Jiang,et al.  A Small-Gain Approach to Robust Event-Triggered Control of Nonlinear Systems , 2015, IEEE Transactions on Automatic Control.

[16]  Richard S. Sutton,et al.  Neuronlike adaptive elements that can solve difficult learning control problems , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[17]  Robert F. Stengel,et al.  An adaptive critic global controller , 2002, Proceedings of the 2002 American Control Conference (IEEE Cat. No.CH37301).

[18]  Snehasis Mukhopadhyay,et al.  To communicate or not to communicate: A decision-theoretic approach to decentralized adaptive control , 2010, Proceedings of the 2010 American Control Conference.

[19]  William B. Dunbar,et al.  Distributed Receding Horizon Control of Dynamically Coupled Nonlinear Systems , 2007, IEEE Transactions on Automatic Control.

[20]  Bruno Castro da Silva,et al.  TD-δπ: a model-free algorithm for efficient exploration , 2012, AAAI 2012.

[21]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[22]  Sarangapani Jagannathan,et al.  Distributed event-sampled approximate optimal control of interconnected affine nonlinear continuous-time systems , 2016, 2016 American Control Conference (ACC).

[23]  Frank L. Lewis,et al.  Neural Network Control Of Robot Manipulators And Non-Linear Systems , 1998 .