Structured Online Learning for Low-Level Control of Quadrotors

Although effective low-level control configurations of quadrotors are already known, the tuning of such controllers requires extensive expert knowledge which can impede their design and deployment. Considering the growing demand for quadrotors in different environments, the importance of an automated approach to designing the controller cannot be neglected. For this purpose, recently, a successful implementation of a model-based reinforcement learning technique was demonstrated by training a neural network using only flight data. In this paper, as an alternative to the neural network approach, we employ a structured model parameterized by a set of bases to identify the governing dynamics of quadrotors. The model accompanied by a value function defined in the product space of the bases leads to an analytical update rule for the controller that can be effectively solved by ODE solvers. The runtime results confirm that the controller together with a recursive least squares identifier can be used as a lightweight framework for learning to stabilize an unknown quadrotor at a given position. In the simulation results, a nonlinear model of the quadrotor is exploited that replaces the real unknown quadrotor. The flight data and 3D graphical simulation are generated to verify the presented learning approach.

[1]  Milad Farsi,et al.  A Structured Online Learning Approach to Nonlinear Tracking with Unknown Dynamics , 2021, 2021 American Control Conference (ACC).

[2]  Jun Liu,et al.  Structured Online Learning-based Control of Continuous-time Nonlinear Systems , 2020, IFAC-PapersOnLine.

[3]  Maximilian Karl,et al.  Learning to Fly via Deep Model-Based Reinforcement Learning , 2020, ArXiv.

[4]  Gabriel Dulac-Arnold,et al.  Challenges of Real-World Reinforcement Learning , 2019, ArXiv.

[5]  Sergey Levine,et al.  Low-Level Control of a Quadrotor With Deep Model-Based Reinforcement Learning , 2019, IEEE Robotics and Automation Letters.

[6]  E Kaiser,et al.  Sparse identification of nonlinear dynamics for model predictive control in the low-data limit , 2017, Proceedings of the Royal Society A.

[7]  Claire J. Tomlin,et al.  Learning quadrotor dynamics using neural network for flight control , 2016, 2016 IEEE 55th Conference on Decision and Control (CDC).

[8]  Jerome Le Ny,et al.  Design of a Trajectory Tracking Controller for a Nanoquadcopter , 2016, ArXiv.

[9]  Pieter Abbeel,et al.  Benchmarking Deep Reinforcement Learning for Continuous Control , 2016, ICML.

[10]  Stefano Bruni,et al.  An efficient recursive least square-based condition monitoring approach for a rail vehicle suspension system , 2016 .

[11]  Jianxiang Xi,et al.  Robust attitude controller design for miniature quadrotors , 2016 .

[12]  Warren E. Dixon,et al.  Model-based reinforcement learning for approximate optimal regulation , 2016, Autom..

[13]  S. Brunton,et al.  Discovering governing equations from data by sparse identification of nonlinear dynamical systems , 2015, Proceedings of the National Academy of Sciences.

[14]  Vladimir Gaitsgory,et al.  Stabilization with discounted optimal control , 2015, Syst. Control. Lett..

[15]  Xiaojun Qiu,et al.  A recursive least square algorithm for active control of mixed noise , 2015 .

[16]  Warren E. Dixon,et al.  Efficient model-based reinforcement learning for approximate online optimal control , 2015, Autom..

[17]  Dragan Nesic,et al.  Stability of infinite-horizon optimal control with discounted cost , 2014, 53rd IEEE Conference on Decision and Control.

[18]  Zhihao Cai,et al.  Self-tuning PID control design for quadrotor UAV based on adaptive pole placement control , 2013, 2013 Chinese Automation Congress.

[19]  Youmin Zhang,et al.  An Efficient Model Predictive Control Scheme for an Unmanned Quadrotor Helicopter , 2013, J. Intell. Robotic Syst..

[20]  Frank L. Lewis,et al.  Reinforcement Learning and Approximate Dynamic Programming for Feedback Control , 2012 .

[21]  M. Schreier,et al.  Modeling and adaptive control of a quadrotor , 2012, 2012 IEEE International Conference on Mechatronics and Automation.

[22]  Ning Zhou,et al.  Probing Signal Design for Power System Identification , 2010, IEEE Transactions on Power Systems.

[23]  Pieter Abbeel,et al.  Apprenticeship learning for helicopter control , 2009, CACM.

[24]  D. Scherer,et al.  VPython: 3D interactive scientific graphics for students , 2000, Comput. Sci. Eng..

[25]  Warren E. Dixon,et al.  Model-Based Reinforcement Learning for Approximate Optimal Control , 2018 .

[26]  Frank L. Lewis,et al.  A novel actor-critic-identifier architecture for approximate optimal control of uncertain nonlinear systems , 2013, Autom..