Using Soft Actor-Critic for Low-Level UAV Control

Unmanned Aerial Vehicles (UAVs), or drones, have recently been used in several civil application domains from organ delivery to remote locations to wireless network coverage. These platforms, however, are naturally unstable systems for which many different control approaches have been proposed. Generally based on classic and modern control, these algorithms require knowledge of the robot's dynamics. However, recently, model-free reinforcement learning has been successfully used for controlling drones without any prior knowledge of the robot model. In this work, we present a framework to train the Soft Actor-Critic (SAC) algorithm to low-level control of a quadrotor in a go-to-target task. All experiments were conducted under simulation. With the experiments, we show that SAC can not only learn a robust policy, but it can also cope with unseen scenarios. Videos from the simulations are available in this https URL and the code in this https URL.

[1]  Sergey Levine,et al.  Learning deep control policies for autonomous aerial vehicles with MPC-guided policy search , 2015, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[2]  Seungwon Choi,et al.  Inverse reinforcement learning control for trajectory tracking of a multirotor UAV , 2017 .

[3]  Yuan Shen,et al.  Autonomous Navigation of UAVs in Large-Scale Complex Environments: A Deep Reinforcement Learning Approach , 2019, IEEE Transactions on Vehicular Technology.

[4]  Azer Bestavros,et al.  Reinforcement Learning for UAV Attitude Control , 2018, ACM Trans. Cyber Phys. Syst..

[5]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[6]  Xu Jie,et al.  Learning to fly , 2019, ACM Trans. Graph..

[7]  Angelo Cangelosi,et al.  Autonomous Quadrotor Landing using Deep Reinforcement Learning , 2017, ArXiv.

[8]  Surya P. N. Singh,et al.  V-REP: A versatile and scalable robot simulation framework , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[9]  Shayegan Omidshafiei.  Reinforcement Learning-based Quadcopter Control , 2015 .

[10]  Roland Siegwart,et al.  Control of a Quadrotor With Reinforcement Learning , 2017, IEEE Robotics and Automation Letters.

[11]  Carlos Sampedro,et al.  Image-Based Visual Servoing Controller for Multirotor Aerial Robots Using Deep Reinforcement Learning , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[12]  Aleksandra Faust,et al.  Air Learning: An AI Research Platform for Algorithm-Hardware Benchmarking of Autonomous Aerial Robots , 2019, ArXiv.

[13]  Zhihong Liu,et al.  Monocular Vision based Autonomous Landing of Quadrotor through Deep Reinforcement Learning , 2018, 2018 37th Chinese Control Conference (CCC).

[14]  Henry Zhu,et al.  Soft Actor-Critic Algorithms and Applications , 2018, ArXiv.

[15]  Guilherme Cano Lopes,et al.  Intelligent Control of a Quadrotor with Proximal Policy Optimization Reinforcement Learning , 2018, 2018 Latin American Robotic Symposium, 2018 Brazilian Symposium on Robotics (SBR) and 2018 Workshop on Robotics in Education (WRE).

[16]  Andres Hernandez,et al.  Identification and path following control of an AR.Drone quadrotor , 2013, 2013 17th International Conference on System Theory, Control and Computing (ICSTCC).

[17]  Sergey Levine,et al.  Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor , 2018, ICML.

[18]  Ashish Kapoor,et al.  AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles , 2017, FSR.

[19]  Weihong Wang,et al.  Vision-based Navigation of UAV with Continuous Action Space Using Deep Reinforcement Learning , 2019, 2019 Chinese Control And Decision Conference (CCDC).

[20]  Andrew J. Davison,et al.  PyRep: Bringing V-REP to Deep Robot Learning , 2019, ArXiv.

[21]  Shaojie Shen,et al.  Learning Unmanned Aerial Vehicle Control for Autonomous Target Following , 2017, IJCAI.