Online integral reinforcement learning control for an uncertain highly flexible aircraft using state and output feedback

Abstract This paper discussed the data-driven control design of a highly flexible aircraft (HFA) with uncertainties. By introducing an integral reinforcement learning (IRL) technique, a novel online model-free control strategy is developed to stabilize the uncertain HFA. Full state feedback with all states measurable and output feedback using an online reinforcement learning scheme to estimate unmeasurable states are considered. With the help of Lyapunov's direct method and under some system assumptions, it is rigorously proved that the proposed IRL based controller can guarantee the asymptotic stability of the closed-loop system. Simulation results show the effectiveness of the proposed scheme.

[1]  Anuradha M. Annaswamy,et al.  On Adaptive Control With Closed-Loop Reference Models: Transients, Oscillations, and Peaking , 2013, IEEE Access.

[2]  Irene Gregory,et al.  Dynamic inversion to control large flexible transport aircraft , 1998 .

[3]  Eric L. Brown,et al.  Integrated strain actuation in aircraft with highly flexible composite wings , 2003 .

[4]  Anuradha M. Annaswamy,et al.  Modeling for Control of Very Flexible Aircraft , 2011 .

[5]  P. Goulart,et al.  Robust Gust Alleviation and Stabilization of Very Flexible Aircraft , 2013 .

[6]  Rafael Palacios,et al.  Re-examined Structural Design Procedures for Very Flexible Aircraft , 2014 .

[7]  Carlos E. S. Cesnik,et al.  Trajectory Control for Very Flexible Aircraft , 2006 .

[8]  Anuradha M. Annaswamy,et al.  Adaptive Output-Feedback Control with Closed-Loop Reference Models for Very Flexible Aircraft , 2016 .

[9]  Ilya Kolmanovsky,et al.  Trajectory Control of Very Flexible Aircraft with Gust Disturbance , 2013 .

[10]  Mayuresh J. Patil,et al.  Flight Control for Flexible, High-Aspect-Ratio Flying Wings , 2010 .

[11]  Yinan Wang,et al.  Nonlinear Aeroelastic Control of Very Flexible Aircraft Using Model Updating , 2018, Journal of Aircraft.

[12]  Joaquim R. R. A. Martins,et al.  Model-Predictive Gust Load Alleviation Controller for a Highly Flexible Aircraft , 2012 .

[13]  Yixin Yin,et al.  Data-Driven Robust Control of Discrete-Time Uncertain Linear Systems via Off-Policy Reinforcement Learning , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[14]  Martin Hromcik,et al.  Active gust load alleviation system for flexible aircraft: Mixed feedforward/feedback approach , 2015 .

[15]  Mayuresh J. Patil,et al.  Nonlinear aeroelastic analysis, flight dynamics, and control of a complete aircraft , 1999 .

[17]  D. Hodges.  A mixed variational formulation based on exact intrinsic equations for dynamics of moving beams , 1990 .

[18]  Anuradha M. Annaswamy,et al.  Shared Control Between Adaptive Autopilots and Human Operators for Anomaly Mitigation , 2019, IFAC-PapersOnLine.

[19]  Andrew Wynn,et al.  A Nonlinear Modal Aeroservoelastic Analysis Framework for Flexible Aircraft , 2016 .

[20]  Derong Liu,et al.  Policy Iteration Adaptive Dynamic Programming Algorithm for Discrete-Time Nonlinear Systems , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[21]  Zhong-Ping Jiang,et al.  Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics , 2012, Autom..

[22]  Frank L. Lewis,et al.  Adaptive optimal control for continuous-time linear systems based on policy iteration , 2009, Autom..

[23]  Xiaowei Zhao,et al.  Preview-Based Altitude Control for a Very Flexible Flying Wing with Lidar Wind Measurements , 2018, 2018 IEEE Conference on Decision and Control (CDC).

[24]  Flavio J. Silvestre,et al.  Gust load alleviation in a flexible smart idealized wing , 2019, Aerospace Science and Technology.

[25]  Guoming G. Zhu,et al.  Smooth-switching LPV control for vibration suppression of a flexible airplane wing , 2019, Aerospace Science and Technology.

[26]  Anuradha M. Annaswamy,et al.  Improved Transient Response in Adaptive Control Using Projection Algorithms and Closed Loop Reference Models , 2012 .

[27]  F. Lewis,et al.  Reinforcement Learning and Feedback Control: Using Natural Decision Methods to Design Optimal Adaptive Controllers , 2012, IEEE Control Systems.

[28]  Frank L. Lewis,et al.  Optimal and Autonomous Control Using Reinforcement Learning: A Survey , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[29]  Ilya Kolmanovsky,et al.  Gust Load Alleviation Control for Very Flexible Aircraft , 2011 .

[30]  Lei Guo,et al.  Disturbance-Observer-Based Control and Related Methods—An Overview , 2016, IEEE Transactions on Industrial Electronics.

[31]  Bo Pang,et al.  Robust Policy Iteration for Continuous-Time Linear Quadratic Regulation , 2020, IEEE Transactions on Automatic Control.

[32]  Carlos E. S. Cesnik,et al.  Nonlinear Flight Dynamics of Very Flexible Aircraft , 2005 .

[33]  Mayuresh J. Patil,et al.  Flight Dynamics of High Aspect-Ratio Flying Wings: Effect of Large Trim Deformation , 2007 .

[34]  Derong Liu,et al.  Value Iteration Adaptive Dynamic Programming for Optimal Control of Discrete-Time Nonlinear Systems , 2016, IEEE Transactions on Cybernetics.

[35]  M Gregory Irene,et al.  Stability Result for Dynamic Inversion Devised to Control Large Flexible Aircraft , 2001 .

[36]  Natsuki Tsushima,et al.  A study on adaptive vibration control and energy conversion of highly flexible multifunctional wings , 2018, Aerospace Science and Technology.

[37]  Anuradha M. Annaswamy,et al.  Robust Adaptive Control , 1984, 1984 American Control Conference.

[38]  Anuradha M. Annaswamy,et al.  Adaptive Control for a Class of Multi-Input Multi-Output Plants With Arbitrary Relative Degree , 2020, IEEE Transactions on Automatic Control.

[39]  R. Bellman.  Dynamic Programming , 1957, Science.

[40]  Carlos E. S. Cesnik,et al.  Nonlinear Aeroelasticity and Flight Dynamics of High-Altitude Long-Endurance Aircraft , 2001 .

[41]  Frank L. Lewis,et al.  Adaptive Suboptimal Output-Feedback Control for Linear Systems Using Integral Reinforcement Learning , 2015, IEEE Transactions on Control Systems Technology.

[42]  Zhou Zhou,et al.  Longitudinal Flight Dynamics and Control of Highly Flexible Solar UAV , 2010, 2010 2nd International Conference on Information Engineering and Computer Science.

[43]  Emanuele Garone,et al.  Explicit Reference Governor for Constrained Maneuver and Shape Control of a Seven-State Multibody Aircraft , 2020 .

[44]  Frank L. Lewis,et al.  Reinforcement Learning for Partially Observable Dynamic Processes: Adaptive Dynamic Programming Using Measured Output Data , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[45]  Carlos E. S. Cesnik,et al.  Dynamic Response of Highly Flexible Flying Wings , 2011 .

[46]  Christopher M. Shearer,et al.  Coupled nonlinear flight dynamics, aeroelasticity, and control of very flexible aircraft. , 2006 .

[47]  Yinan Wang,et al.  Model-Predictive Control of Flexible Aircraft using Nonlinear Reduced-Order Models , 2016 .

[48]  Q. Wei,et al.  Off-Policy Integral Reinforcement Learning Method for Multi-player Non-zero-Sum Games , 2018, Studies in Systems, Decision and Control.

[49]  Yan Wan,et al.  Adaptive Optimal Control for Stochastic Multiplayer Differential Games Using On-Policy and Off-Policy Reinforcement Learning , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[50]  Dewey H. Hodges,et al.  Flight Dynamics of Highly Flexible Flying Wings , 2006 .

[51]  T. Livet,et al.  PARAMETER ROBUST FLIGHT CONTROL SYSTEM FOR A FLEXIBLE AIRCRAFT , 1994 .

[52]  Derong Liu,et al.  Discrete-Time Optimal Control via Local Policy Iteration Adaptive Dynamic Programming , 2017, IEEE Transactions on Cybernetics.

[53]  D. Kleinman.  On an iterative technique for Riccati equation computations , 1968 .

[54]  Sarangapani Jagannathan,et al.  Online Optimal Control of Affine Nonlinear Discrete-Time Systems With Unknown Internal Dynamics by Using Time-Based Policy Update , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[55]  Pedro Paglione,et al.  Aircraft Control Based on Flexible Aircraft Dynamics , 2017 .

[56]  Jonathan E. Cooper,et al.  LQG based model predictive control for gust load alleviation , 2017 .

[57]  Dewey H. Hodges,et al.  Output Feedback Control of the Nonlinear Aeroelastic Response of a Slender Wing , 2002 .

[58]  Irene M. Gregory,et al.  Modified dynamic inversion to control large flexible aircraft - What's going on? , 1999 .