Morphing control of a new bionic morphing UAV with deep reinforcement learning

Abstract With rapid development of aviation technology, materials science and artificial intelligence, aircraft design is pursuing higher requirements both in civil and military fields. The new generation of aircraft should have the autonomous capable of performing a variety of tasks (such as take-off and landing, cruising, maneuvering, hover, attack, etc.) under a highly variable flight environment (height, Mach number, etc.) and meanwhile maintaining good performance. Morphing aircraft can use smart materials and actuators to autonomously deform the shape according to the changes in flight environment and mission, and always maintain an optimal aerodynamic shape, therefore get flourished developments. Based on the ability of birds to stretch wings when flying at low speed and to constrict wings at high speed, a new bionic morphing UAV has been designed and developed as the study model by our team. In order to make this new aircraft be able to complete rapid autonomous morphing and aerodynamic performance optimization under different missions and flight conditions, we developed deep neural networks and reinforcement learning techniques as a control strategy. Considering the continuity of the state and action spaces for model, the Deep Deterministic Policy Gradient (DDPG) algorithm based on the actor-critic, model-free algorithm was adopted and verified on the classic nonlinear Pendulum model and Cart Pole game. After the feasibility was verified, morphing aircraft model was controlled to complete prescribed deformation using DDPG algorithm. Furthermore, on the condition that the DDPG algorithm can control morphing well, through training and testing on model using simulation data from wind tunnel tests and actual flight, the autonomous morphing control for the shape optimization of the bionic morphing UAV model could be realized.

[1]  Guoming G. Zhu,et al.  Optimum distributed wing shaping and control loads for highly flexible aircraft , 2018, Aerospace Science and Technology.

[2]  Wei Hu,et al.  Exploring Deep Reinforcement Learning with Multi Q-Learning , 2016 .

[3]  Sergey Levine,et al.  Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[4]  John Valasek,et al.  Reinforcement Learning of a Morphing Airfoil-Policy and Discrete Learning Analysis , 2008, J. Aerosp. Comput. Inf. Commun..

[5]  Guy Lever,et al.  Deterministic Policy Gradient Algorithms , 2014, ICML.

[6]  Mark R. Cutkosky,et al.  Wings of a Feather Stick Together: Morphing Wings with Barbule-Inspired Latching , 2015, Living Machines.

[7]  Jayanth N. Kudva,et al.  Development of Next Generation Morphing Aircraft Structures , 2007 .

[8]  Jayanth N. Kudva,et al.  Morphing aircraft concepts, classifications, and challenges , 2004, SPIE Smart Structures and Materials + Nondestructive Evaluation and Health Monitoring.

[9]  Pedro B. Leal,et al.  Shape memory alloy-based mechanism for aeronautical application: Theory, optimization and experiment , 2018 .

[10]  Peter Stone,et al.  Deep Recurrent Q-Learning for Partially Observable MDPs , 2015, AAAI Fall Symposia.

[11]  M Di Luca,et al.  Bioinspired morphing wings for extended flight envelope and roll control of small drones , 2017, Interface Focus.

[12]  Inderjit Chopra,et al.  Review of State of Art of Smart Structures and Integrated Systems , 2002 .

[13]  Rick Lind,et al.  Time-varying dynamics of a micro air vehicle with variable-sweep morphing , 2009 .

[14]  Kjell Kersandt,et al.  Deep reinforcement learning as control method for autonomous UAVs , 2018 .

[15]  Darren J. Hartl,et al.  Control of Morphing Wing Shapes with Deep Reinforcement Learning , 2018 .

[16]  Michael I. Friswell,et al.  Effect of symmetric and asymmetric span morphing on flight dynamics , 2014 .

[17]  Yishay Mansour,et al.  Policy Gradient Methods for Reinforcement Learning with Function Approximation , 1999, NIPS.

[18]  I-Ming Chen,et al.  Autonomous navigation of UAV by using real-time model-based reinforcement learning , 2016, 2016 14th International Conference on Control, Automation, Robotics and Vision (ICARCV).

[19]  Pedro Gamboa,et al.  Design Optimization of a Variable-Span Morphing Wing for , 2011 .

[20]  Hussein A. Abbass,et al.  Multi-Task Deep Reinforcement Learning for Continuous Action Control , 2017, IJCAI.

[21]  Tom Schaul,et al.  Dueling Network Architectures for Deep Reinforcement Learning , 2015, ICML.

[22]  Samantha Hansen,et al.  Using Deep Q-Learning to Control Optimization Hyperparameters , 2016, ArXiv.

[23]  Kamesh Subbarao,et al.  Modeling of Flight Dynamics of Morphing Wing Aircraft , 2011 .

[24]  Terrence A. Weisshaar,et al.  Evaluating the Impact of Morphing Technologies on Aircraft Performance , 2002 .

[25]  Ramesh Raskar,et al.  Designing Neural Network Architectures using Reinforcement Learning , 2016, ICLR.

[26]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[27]  Daniel J. Inman,et al.  A Review of Morphing Aircraft , 2011 .

[28]  Nikhil Nigam,et al.  Adaptive Control and Actuation System Development for Biomimetic Morphing , 2016 .

[29]  John Valasek,et al.  Reinforcement Learning of Morphing Airfoils with Aerodynamic and Structural Effects , 2007, J. Aerosp. Comput. Inf. Commun..

[30]  Rafic M. Ajaj,et al.  The Transformer aircraft: A multimission unmanned aerial vehicle capable of symmetric and asymmetric span morphing , 2018 .

[31]  Monish D. Tandale,et al.  Improved Adaptive–Reinforcement Learning Control for Morphing Unmanned Air Vehicles , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[32]  Joaquim R. R. A. Martins,et al.  Design of a transonic wing with an adaptive morphing trailing edge via aerostructural optimization , 2018, Aerospace Science and Technology.

[33]  John Valasek,et al.  Morphing Airfoils with Four Morphing Parameters , 2008 .

[34]  Monish D. Tandale,et al.  A Reinforcement Learning - Adaptive Control Architecture for Morphing , 2004, J. Aerosp. Comput. Inf. Commun..

[35]  Mostafa Hassanalian,et al.  Morphing and growing micro unmanned air vehicle: Sizing process and stability , 2018, Aerospace Science and Technology.

[36]  Mujahid Abdulrahim,et al.  Using Avian Morphology To Enhance Aircraft Maneuverability , 2006 .