Extremum Seeking for Creating Optimal Feedback Controls of Unknown Systems by Tuning Basis Functions

We consider the problem of optimal feedback control of unknown dynamic systems. Optimal feedback control is important for systems with time-varying initial conditions, which cannot rely on feedforward controllers. For example, components in particle accelerators are turned on and off hundreds of times per second, with pulse widths of ∼1 ms and repetition rates >100 Hz, whose dynamics and initial conditions vary slowly over time due to external disturbances, such as temperature fluctuations. Our approach allows us to quickly learn an optimal feedback control, which can then be applied for all initial conditions even as the system begins to drift and change. For linear systems we reproduce the cost minimizing linear quadratic regulator (LQR) optimal controller that could have been designed had the system been known.

[1]  M. Phan,et al.  Linear quadratic optimal learning control (LQL) , 2000 .

[2]  Zhiyong Sun,et al.  Formation Shape Control Based on Distance Measurements Using Lie Bracket Approximations , 2018, SIAM J. Control. Optim..

[3]  Mouhacine Benosman,et al.  Multi‐parametric extremum seeking‐based iterative feedback gains tuning for nonlinear control , 2016 .

[4]  Christian Ebenbauer,et al.  On a class of generating vector fields for the extremum seeking problem: Lie bracket approximation and stability properties , 2017, Autom..

[5]  Sanjoy K. Mitter,et al.  Conjugate convex functions, duality, and optimal control problems I: Systems governed by ordinary differential equations , 1970, Information Sciences.

[6]  A. Scheinker.  Bounded extremum seeking for angular velocity actuated control of nonholonomic unicycle , 2017 .

[7]  Zhong-Ping Jiang,et al.  Output-feedback adaptive optimal control of interconnected systems based on robust adaptive dynamic programming , 2016, Autom..

[8]  Alexander Scheinker,et al.  Application of Extremum Seeking for Time-Varying Systems to Resonance Control of RF Cavities , 2017, IEEE Transactions on Control Systems Technology.

[9]  Frank L. Lewis,et al.  Adaptive optimal control for continuous-time linear systems based on policy iteration , 2009, Autom..

[10]  Milos S. Stankovic,et al.  Lie bracket approximation of extremum seeking systems , 2011, Autom..

[11]  Simon Michalowsky,et al.  Swinging up the Stephenson-Kapitza pendulum , 2013, 52nd IEEE Conference on Decision and Control.

[12]  Alexander Scheinker,et al.  Simultaneous stabilization and optimization of unknown, time-varying systems , 2013, 2013 American Control Conference.

[13]  Thomas G. Fisher.  Batch Control Systems: Design, Application, and Implementation , 1990 .

[14]  Mark Haring,et al.  Extremum-seeking control for nonlinear systems with periodic steady-state outputs , 2013, Autom..

[15]  Frank L. Lewis,et al.  Reinforcement Learning for Partially Observable Dynamic Processes: Adaptive Dynamic Programming Using Measured Output Data , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[16]  Frank L. Lewis,et al.  Optimal tracking control of nonlinear partially-unknown constrained-input systems using integral reinforcement learning , 2014, Autom..

[17]  Miroslav Krstic,et al.  Extremum seeking with bounded update rates , 2014, Syst. Control. Lett..

[18]  Kevin L. Moore,et al.  Iterative Learning Control: Brief Survey and Categorization , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[19]  R. Rockafellar.  Integrals which are convex functionals. II , 1968 .

[20]  Xiang Li,et al.  Iterative learning impedance control for rehabilitation robots driven by series elastic actuators , 2018, Autom..

[21]  M. Yamakita,et al.  Adaptive output optimal control algorithm for unknown system dynamics based on policy iteration , 2010, Proceedings of the 2010 American Control Conference.

[22]  Alexander Scheinker,et al.  Constrained extremum seeking stabilization of systems not affine in control , 2018 .

[23]  Denis Dochain,et al.  Flatness-Based Extremum-Seeking Control Over Periodic Orbits , 2007, IEEE Transactions on Automatic Control.

[24]  J. Y. Choi,et al.  Adaptive iterative learning control of uncertain robotic systems , 2000 .

[25]  Alexander Scheinker,et al.  Bounded extremum seeking with discontinuous dithers , 2016, Autom..

[26]  Xin Zhang,et al.  Data-Driven Robust Approximate Optimal Tracking Control for Unknown General Nonlinear Systems Using Adaptive Dynamic Programming Method , 2011, IEEE Transactions on Neural Networks.

[27]  Qinmin Yang,et al.  Reinforcement Learning Controller Design for Affine Nonlinear Discrete-Time Systems using Online Approximators , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[28]  Raik Suttner.  Extremum seeking control with an adaptive dither signal , 2019, Autom..

[29]  A.G. Alleyne,et al.  A survey of iterative learning control , 2006, IEEE Control Systems.

[30]  Suguru Arimoto,et al.  Bettering operation of Robots by learning , 1984, J. Field Robotics.

[31]  Zhong-Ping Jiang,et al.  Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics , 2012, Autom..

[32]  Eugenio Schuster,et al.  Mixing enhancement in 2D magnetohydrodynamic channel flow by extremum seeking boundary control , 2009, 2009 American Control Conference.

[33]  Frank L. Lewis,et al.  Reinforcement Q-learning for optimal tracking control of linear discrete-time systems with unknown dynamics , 2014, Autom..

[34]  Miroslav Krstic,et al.  Iterative learning control based on extremum seeking , 2016, Autom..

[35]  Sarangapani Jagannathan,et al.  Optimal control of unknown affine nonlinear discrete-time systems using offline-trained neural networks with proof of convergence , 2009, Neural Networks.

[36]  Qinglai Wei,et al.  Optimal control of unknown nonaffine nonlinear discrete-time systems based on adaptive dynamic programming , 2012, Autom..

[37]  Miroslav Krstic,et al.  Power Optimization for Photovoltaic Microconverters Using Multivariable Newton-Based Extremum Seeking , 2012, IEEE Transactions on Control Systems Technology.

[38]  Gökhan M. Atinç,et al.  Extremum seeking-based adaptive control for electromagnetic actuators , 2015, Int. J. Control.

[39]  Christian Ebenbauer,et al.  On extremum seeking controllers based on the Lie bracket approximation in domains with obstacles , 2018, PAMM.

[40]  Miroslav Krstic,et al.  Batch-to-Batch Finite-Horizon LQ Control for Unknown Discrete-Time Linear Systems Via Stochastic Extremum Seeking , 2017, IEEE Transactions on Automatic Control.

[41]  Miroslav Krstic,et al.  Finite-horizon LQ control for unknown discrete-time linear systems via extremum seeking , 2012, 2012 IEEE 51st IEEE Conference on Decision and Control (CDC).

[42]  M. Walker,et al.  Design and simulation of extremum-seeking open-loop optimal control of current profile in the DIII-D tokamak , 2008 .

[43]  Simon Michalowsky,et al.  A family of extremum seeking laws for a unicycle model with a moving target: theoretical and experimental studies , 2018, 2018 European Control Conference (ECC).

[44]  Denis Dochain,et al.  A proportional-integral extremum-seeking controller design technique , 2017, Autom..

[45]  Emanuele Garone,et al.  Newton-based extremum seeking: A second-order Lie bracket approximation approach , 2019, Autom..