Discrete-Time Impulsive Adaptive Dynamic Programming

In this paper, a new iterative adaptive dynamic programming (ADP) algorithm is developed to solve optimal impulsive control problems for infinite horizon discrete-time nonlinear systems. Considering the constraint of the impulsive interval, in each iteration, the iterative impulsive value function under each possible impulsive interval is obtained, and then the iterative value function and iterative control law are achieved. A new convergence analysis method is developed which proves an iterative value function to converge to the optimum as the iteration index increases to infinity. The properties of the iterative control law are analyzed, and the detailed implementation of the optimal impulsive control law is presented. Finally, two simulation examples with comparisons are given to show the effectiveness of the developed method.

[1]  Yu Zhang,et al.  Impulsive Control of Discrete Systems With Time Delay , 2009, IEEE Transactions on Automatic Control.

[2]  Athanasios V. Vasilakos,et al.  Differential Evolution With Event-Triggered Impulsive Control , 2015, IEEE Transactions on Cybernetics.

[3]  Dongbin Zhao,et al.  Comprehensive comparison of online ADP algorithms for continuous-time optimal control , 2017, Artificial Intelligence Review.

[4]  A. Rantzer.  Relaxed dynamic programming in switching systems , 2006 .

[5]  Xiaodi Li,et al.  Stabilization of Delay Systems: Delay-Dependent Impulsive Control , 2017, IEEE Transactions on Automatic Control.

[6]  Frank L. Lewis,et al.  Discrete-Time Nonlinear HJB Solution Using Approximate Dynamic Programming: Convergence Proof , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[7]  Yu Liu,et al.  Optimal constrained self-learning battery sequential management in microgrid via adaptive dynamic programming , 2017, IEEE/CAA Journal of Automatica Sinica.

[8]  Derong Liu,et al.  Online Synchronous Approximate Optimal Learning Algorithm for Multi-Player Non-Zero-Sum Games With Unknown Dynamics , 2014, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[9]  Xiong Yang,et al.  Adaptive Critic Designs for Event-Triggered Robust Control of Nonlinear Systems With Unknown Dynamics , 2019, IEEE Transactions on Cybernetics.

[10]  Qinglai Wei,et al.  Discrete-Time Stable Generalized Self-Learning Optimal Control With Approximation Errors , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[11]  Chuandong Li,et al.  On Hybrid Impulsive and Switching Neural Networks , 2008, IEEE Trans. Syst. Man Cybern. Part B.

[12]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[13]  Frank L. Lewis,et al.  Adaptive Optimal Control of Unknown Constrained-Input Systems Using Policy Iteration and Neural Networks , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[14]  Haibo He,et al.  Model-Free Adaptive Control for Unknown Nonlinear Zero-Sum Differential Game , 2018, IEEE Transactions on Cybernetics.

[15]  Derong Liu,et al.  Policy Iteration Adaptive Dynamic Programming Algorithm for Discrete-Time Nonlinear Systems , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[16]  Qichao Zhang,et al.  Experience Replay for Optimal Control of Nonzero-Sum Game Systems With Unknown Dynamics , 2016, IEEE Transactions on Cybernetics.

[17]  Ali Heydari,et al.  Fixed-final-time optimal tracking control of input-affine nonlinear systems , 2014, Neurocomputing.

[18]  Huaguang Zhang,et al.  Distributed Cooperative Optimal Control for Multiagent Systems on Directed Graphs: An Inverse Optimal Approach , 2015, IEEE Transactions on Cybernetics.

[19]  Haibo He,et al.  Online Learning Control Using Adaptive Critic Designs With Sparse Kernel Machines , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[20]  Frank L. Lewis,et al.  Discrete-Time Local Value Iteration Adaptive Dynamic Programming: Convergence Analysis , 2018, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[21]  Derong Liu,et al.  Data-Based Adaptive Critic Designs for Nonlinear Robust Optimal Control With Uncertain Dynamics , 2016, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[22]  Huaguang Zhang,et al.  Neural-Network-Based Near-Optimal Control for a Class of Discrete-Time Affine Nonlinear Systems With Control Constraints , 2009, IEEE Transactions on Neural Networks.

[23]  Frank L. Lewis,et al.  Error-Tolerant Iterative Adaptive Dynamic Programming for Optimal Renewable Home Energy Scheduling and Battery Management , 2017, IEEE Transactions on Industrial Electronics.

[24]  Yang Liu,et al.  Controllability for a Class of Linear Time-Varying Impulsive Systems With Time Delay in Control Input , 2011, IEEE Transactions on Automatic Control.

[25]  Huaguang Zhang,et al.  Novel Weighting-Delay-Based Stability Criteria for Recurrent Neural Networks With Time-Varying Delay , 2010, IEEE Transactions on Neural Networks.

[26]  Huaguang Zhang,et al.  Adaptive Dynamic Programming for a Class of Complex-Valued Nonlinear Systems , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[27]  Dimitri P. Bertsekas,et al.  Value and Policy Iterations in Optimal Control and Adaptive Dynamic Programming , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[28]  Huaguang Zhang,et al.  Near-Optimal Control for Nonzero-Sum Differential Games of Continuous-Time Nonlinear Systems Using Single-Network ADP , 2013, IEEE Transactions on Cybernetics.

[29]  Sérgio Loureiro Fraga,et al.  Hamilton-Jacobi-Bellman Equation and Feedback Synthesis for Impulsive Control , 2012, IEEE Transactions on Automatic Control.

[30]  Haibo He,et al.  Functional Nonlinear Model Predictive Control Based on Adaptive Dynamic Programming , 2019, IEEE Transactions on Cybernetics.

[31]  Frank L. Lewis,et al.  Actor–Critic-Based Optimal Tracking for Partially Unknown Nonlinear Discrete-Time Systems , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[32]  Guangming Xie,et al.  Necessary and sufficient conditions for controllability and observability of switched impulsive control systems , 2004, IEEE Transactions on Automatic Control.

[33]  Zhimin He,et al.  Monotone iterative technique for first order impulsive difference equations with periodic boundary conditions , 2004, Appl. Math. Comput..

[34]  Frank L. Lewis,et al.  Optimal tracking control of nonlinear partially-unknown constrained-input systems using integral reinforcement learning , 2014, Autom..

[35]  Haibo He,et al.  Model-Free Dual Heuristic Dynamic Programming , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[36]  Derong Liu,et al.  Decentralized Stabilization for a Class of Continuous-Time Nonlinear Interconnected Systems Using Online Learning Optimal Control Approach , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[37]  Huaguang Zhang,et al.  Robust Global Exponential Synchronization of Uncertain Chaotic Delayed Neural Networks via Dual-Stage Impulsive Control , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[38]  Frank L. Lewis,et al.  Off-Policy Actor-Critic Structure for Optimal Control of Unknown Systems With Disturbances , 2016, IEEE Transactions on Cybernetics.

[39]  Hao Xu,et al.  Neural Network-Based Finite Horizon Stochastic Optimal Control Design for Nonlinear Networked Control Systems , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[40]  Zhong-Ping Jiang,et al.  Global Adaptive Dynamic Programming for Continuous-Time Nonlinear Systems , 2013, IEEE Transactions on Automatic Control.

[41]  Richard S. Sutton,et al.  A Menu of Designs for Reinforcement Learning Over Time , 1995 .

[42]  Frank L. Lewis,et al.  Mixed Iterative Adaptive Dynamic Programming for Optimal Battery Energy Control in Smart Residential Microgrids , 2017, IEEE Transactions on Industrial Electronics.

[43]  Huaguang Zhang,et al.  A Novel Infinite-Time Optimal Tracking Control Scheme for a Class of Discrete-Time Nonlinear Systems via the Greedy HDP Iteration Algorithm , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[44]  Haibo He,et al.  Fuzzy-Based Goal Representation Adaptive Dynamic Programming , 2016, IEEE Transactions on Fuzzy Systems.

[45]  Derong Liu,et al.  Value Iteration Adaptive Dynamic Programming for Optimal Control of Discrete-Time Nonlinear Systems , 2016, IEEE Transactions on Cybernetics.

[46]  Jennie Si,et al.  Online learning control by association and reinforcement. , 2001, IEEE transactions on neural networks.

[47]  Derong Liu,et al.  Adaptive Dynamic Programming for Discrete-Time Zero-Sum Games , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[48]  Huaguang Zhang,et al.  Iterative ADP learning algorithms for discrete-time multi-player games , 2018, Artificial Intelligence Review.

[49]  Frank L. Lewis,et al.  Discrete-Time Deterministic $Q$ -Learning: A Novel Convergence Analysis , 2017, IEEE Transactions on Cybernetics.

[50]  Derong Liu,et al.  Discrete-Time Optimal Control via Local Policy Iteration Adaptive Dynamic Programming , 2017, IEEE Transactions on Cybernetics.

[51]  Ali Heydari,et al.  Feedback Solution to Optimal Switching Problems With Switching Cost , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[52]  Derong Liu,et al.  Residential energy scheduling for variable weather solar energy based on adaptive dynamic programming , 2018, IEEE/CAA Journal of Automatica Sinica.

[53]  Jinyu Wen,et al.  Adaptive Learning in Tracking Control Based on the Dual Critic Network Design , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[54]  F. Lewis,et al.  Reinforcement Learning and Feedback Control: Using Natural Decision Methods to Design Optimal Adaptive Controllers , 2012, IEEE Control Systems.

[55]  Kyriakos G. Vamvoudakis,et al.  Asymptotically Stable Adaptive–Optimal Control Algorithm With Saturating Actuators and Relaxed Persistence of Excitation , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[56]  Yang Xiong,et al.  Adaptive Dynamic Programming with Applications in Optimal Control , 2017 .

[57]  Xiaodi Li,et al.  Impulsive Control for Existence, Uniqueness, and Global Stability of Periodic Solutions of Recurrent Neural Networks With Discrete and Continuously Distributed Delays , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[58]  Huaguang Zhang,et al.  Dissipativity Analysis and Synthesis for a Class of T–S Fuzzy Descriptor Systems , 2017, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[59]  Wei Xing Zheng,et al.  Generating Globally Stable Periodic Solutions of Delayed Neural Networks With Periodic Coefficients via Impulsive Control , 2017, IEEE Transactions on Cybernetics.

[60]  Bo Lincoln,et al.  Relaxing dynamic programming , 2006, IEEE Transactions on Automatic Control.