Modified Q-Learning Algorithm with Lifting Method for Discrete-Time Linear Periodic Systems

This paper presents a lifting technique based modified Q-Learning algorithm for dealing with the optimal control problem of discrete-time linear periodic systems. The algorithm uses a lifting method to convert the periodic LQR problem to the time-invariant LQR problem of an improved system. The novel model-free Q-learning is proposed based on the improved system, and its convergence is proven. The proposed algorithm is implemented online to find optimal solution only using the collected data along the system trajectories. The effectiveness and convergence of the proposed strategies are validated through simulation studies.

[1]  C. Sidney Burrus,et al.  A unified analysis of multirate and periodically time-varying digital filters , 1975 .

[2]  S. Bittanti,et al.  The Riccati equation , 1991 .

[3]  O. Grasselli,et al.  Pole placement for nonreachable periodic discrete-time systems , 1991 .

[4]  A. Laub,et al.  Numerical solution of the discrete-time periodic Riccati equation , 1994, IEEE Trans. Autom. Control..

[5]  Andrew G. Barto,et al.  Adaptive linear quadratic control using policy iteration , 1994, Proceedings of 1994 American Control Conference - ACC '94.

[6]  Tomas Landelius,et al.  Reinforcement Learning and Distributed Local Model Synthesis , 1997 .

[7]  Frank L. Lewis,et al.  Model-free Q-learning designs for linear discrete-time zero-sum games with application to H-infinity control , 2007, Autom..

[8]  F. Lewis,et al.  Model-free Q-learning designs for discrete-time zero-sum games with application to H-infinity control , 2007, 2007 European Control Conference (ECC).

[9]  Pierre Rouchon,et al.  Periodic input estimation for linear periodic systems: Automotive engine applications , 2007, Autom..

[10]  F.L. Lewis,et al.  Reinforcement learning and adaptive dynamic programming for feedback control , 2009, IEEE Circuits and Systems Magazine.

[11]  Frank L. Lewis,et al.  Adaptive optimal control for continuous-time linear systems based on policy iteration , 2009, Autom..

[12]  Matthew S. Allen,et al.  Output-only modal analysis of linear time-periodic systems with application to wind turbine simulation data , 2011 .

[13]  Yaguang Yang,et al.  An Efficient Algorithm for Periodic Riccati Equation for Spacecraft Attitude Control Using Magnetic Torques , 2016 .

[14]  Yaguang Yang.  An efficient LQR design for discrete-time linear periodic system based on a novel lifting method , 2018, Autom..

[15]  Derong Liu,et al.  Adaptive Dynamic Programming for Discrete-Time Zero-Sum Games , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[16]  Frank L. Lewis,et al.  Off-Policy Interleaved $Q$ -Learning: Optimal Control for Affine Nonlinear Discrete-Time Systems , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[17]  Zongli Lin,et al.  Output Feedback Q-Learning Control for the Discrete-Time Linear Quadratic Regulator Problem , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[18]  Tianyou Chai,et al.  Optimal Output Regulation of Linear Discrete-Time Systems With Unknown Dynamics Using Reinforcement Learning , 2020, IEEE Transactions on Cybernetics.

[19]  Guang-Hong Yang,et al.  Adaptive Fuzzy Output Feedback Fault-Tolerant Compensation for Uncertain Nonlinear Systems With Infinite Number of Time-Varying Actuator Failures and Full-State Constraints , 2019, IEEE Transactions on Cybernetics.