Online Series-Parallel Reinforcement-Learning- Based Balancing Control for Reaction Wheel Bicycle Robots on a Curved Pavement

The reaction wheel bicycle robot is a kind of unmanned mobile robot with great potential. However, the control of such bicycle robots on a curved pavement under inaccurate model parameters, model uncertainties and disturbances is challenging due to the lateral instability and underactuated characteristic. Applying conventional control methods to this problem often results in brittle and inaccurate controllers. In this paper, an online serial-parallel combination reinforcement learning with conventional control methods is designed to achieve the path tracking and banlancing control for a reaction wheel bicycle robot on curved pavements. The parallel part of the controller refers to compensating the equilibrium point and the serial part of the controller refers to adjusting the parameters of a sliding mode controller that tracks the target roll equilibrium point. The comparison between the proposed controller and several existing controllers in experimental test built in Matlab Simscape illustrates stronger robustness and better control performances.

[1]  Jiuxiang Dong,et al.  Fuzzy Control Based on Reinforcement Learning and Subsystem Error Derivatives for Strict-Feedback Systems With an Observer , 2023, IEEE Transactions on Fuzzy Systems.

[2]  Jiuxiang Dong,et al.  Performance-constrained fault-tolerant DSC based on reinforcement learning for nonlinear systems with uncertain parameters , 2023, Appl. Math. Comput..

[3]  Xianjin Zhu,et al.  Online Reinforcement-Learning-Based Adaptive Terminal Sliding Mode Control for Disturbed Bicycle Robots on a Curved Pavement , 2022, Electronics.

[4]  Prof. (Dr) Suhas Sakharam Mohite,et al.  Design, analysis and implementation of control moment gyroscope (CMG) mechanism to self-balance a moped bike , 2022, Materials Today: Proceedings.

[5]  Kanghui He,et al.  Learning-Based Trajectory Tracking and Balance Control for Bicycle Robots With a Pendulum: A Gaussian Process Approach , 2022, IEEE/ASME Transactions on Mechatronics.

[6]  F. Ibrahim,et al.  Reinforcement Learning Approach to Autonomous PID Tuning , 2022, 2022 American Control Conference (ACC).

[7]  Haotian Zhao,et al.  Fuzzy model‐based multi‐objective dynamic programming with modified particle swarm optimization approach for the balance control of bicycle robot , 2021, IET Control Theory & Applications.

[8]  Bin Liang,et al.  Continuous reinforcement learning based ramp jump control for single-track two-wheeled robots , 2021, Trans. Inst. Meas. Control.

[9]  Xudong Zheng,et al.  Natural Residual Reinforcement Learning for Bicycle Robot Control , 2021, 2021 IEEE International Conference on Mechatronics and Automation (ICMA).

[10]  Guang-Hong Yang,et al.  Optimal Transmission Power Scheduling of Networked Control Systems Via Fuzzy Adaptive Dynamic Programming , 2021, IEEE Transactions on Fuzzy Systems.

[11]  Hai Wang,et al.  Robust control of reaction wheel bicycle robot via adaptive integral terminal sliding mode , 2021, Nonlinear Dynamics.

[12]  Sepp Hochreiter,et al.  Convergence Proof for Actor-Critic Methods Applied to PPO and RUDDER , 2020, Trans. Large Scale Data Knowl. Centered Syst..

[13]  Lorenz Wellhausen,et al.  Learning quadrupedal locomotion over challenging terrain , 2020, Science Robotics.

[14]  Chunquan Xu,et al.  A Speed Regulating Controller for Virtual Passive Walking , 2020, 2020 IEEE International Conference on Mechatronics and Automation (ICMA).

[15]  Kanokvate Tungpimolrut,et al.  Development and control of a bicycle robot based on steering and pendulum balancing , 2020 .

[16]  Ignacio Carlucho,et al.  An adaptive deep reinforcement learning approach for MIMO PID control of mobile robots. , 2020, ISA transactions.

[17]  Szil'ard Aradi,et al.  Survey of Deep Reinforcement Learning for Motion Planning of Autonomous Vehicles , 2020, IEEE Transactions on Intelligent Transportation Systems.

[18]  Hualong Xie,et al.  A new virtual-real gravity compensated inverted pendulum model and ADAMS simulation for biped robot with heterogeneous legs , 2020, Journal of Mechanical Science and Technology.

[19]  Dariusz Horla,et al.  Introduction of Feedback Linearization to Robust LQR and LQI Control – Analysis of Results from an Unmanned Bicycle Robot with Reaction Wheel , 2019 .

[20]  TaeChoong Chung,et al.  Toward Self-Driving Bicycles Using State-of-the-Art Deep Reinforcement Learning Algorithms , 2019, Symmetry.

[21]  Sergey Levine,et al.  Residual Reinforcement Learning for Robot Control , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[22]  Mingguo Zhao,et al.  Steering Control for Autonomously Balancing Bicycle at Low Speed , 2018, 2018 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[23]  Kate Saenko,et al.  Hierarchical Reinforcement Learning with Hindsight , 2018, ArXiv.

[24]  Makoto Araki,et al.  Study of Riding Assist Control Enabling Self-Standing in Stationary State , 2018, SAE International Journal of Vehicle Dynamics, Stability, and NVH.

[25]  Kate Saenko,et al.  Learning Multi-Level Hierarchies with Hindsight , 2017, ICLR.

[26]  Alec Radford,et al.  Proximal Policy Optimization Algorithms , 2017, ArXiv.

[27]  F. J. Castillo-Garcia,et al.  Improving Energy Efficiency of an Autonomous Bicycle with Adaptive Controller Design , 2017 .

[28]  Mingguo Zhao,et al.  Simultaneous localization and mapping for autonomous bicycles , 2017 .

[29]  Qinglai Wei,et al.  Data-Driven Zero-Sum Neuro-Optimal Control for a Class of Continuous-Time Unknown Nonlinear Systems With Disturbance Using ADP , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[30]  Mingguo Zhao,et al.  Constant-velocity steering control design for unmanned bicycles , 2015, 2015 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[31]  H. Eric Tseng,et al.  State of the art survey: active and semi-active suspension control , 2015 .

[32]  Kiattisin Kanjanawanishkul,et al.  LQR and MPC controller design and comparison for a stationary self-balancing bicycle robot with a reaction wheel , 2015, Kybernetika.

[33]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[34]  Michael I. Jordan,et al.  Trust Region Policy Optimization , 2015, ICML.

[35]  Javier Moreno-Valenzuela,et al.  On trajectory tracking control of the inertia wheel pendulum , 2014, 2014 International Conference on Control, Decision and Information Technologies (CoDIT).

[36]  Keith Redmill,et al.  Gyroscopic stabilization of an unmanned bicycle , 2014, 2014 American Control Conference.

[37]  Hyun-Woo Kim,et al.  Balancing control of bicycle robot using PID control , 2013, 2013 13th International Conference on Control, Automation and Systems (ICCAS 2013).

[38]  Min-Sung Kim,et al.  Balancing and Driving Control of a Bicycle Robot , 2012 .

[39]  Yizhai Zhang,et al.  Balance control and analysis of stationary riderless motorcycles , 2011, 2011 IEEE International Conference on Robotics and Automation.

[40]  Chih-Lyang Hwang,et al.  Fuzzy Sliding-Mode Underactuated Control for Autonomous Dynamic Balance of an Electrical Bicycle , 2009, IEEE Transactions on Control Systems Technology.

[41]  Sebastian Thrun,et al.  Stanley: The robot that won the DARPA Grand Challenge , 2006, J. Field Robotics.

[42]  Dezhen Song,et al.  Trajectory tracking and balance stabilization control of autonomous motorcycles , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[43]  Peter I. Corke,et al.  Nonlinear control of the Reaction Wheel Pendulum , 2001, Autom..

[44]  Preben Alstrøm,et al.  Learning to Drive a Bicycle Using Reinforcement Learning and Shaping , 1998, ICML.

[45]  Jerrold E. Marsden,et al.  Control for an autonomous bicycle , 1995, Proceedings of 1995 IEEE International Conference on Robotics and Automation.

[46]  Tianshu Wang,et al.  Dynamic modeling of an unmanned motorcycle and combined balance control with both steering and double CMGs , 2022, Mechanism and Machine Theory.

[47]  B. You,et al.  Velocity following control of a pseudo-driven wheel for reducing internal forces between wheels , 2022, IEEE Robotics and Automation Letters.

[48]  Chih-Hui Chiu,et al.  Bicycle Robot Balance Control Based on a Robust Intelligent Controller , 2020, IEEE Access.

[49]  Payam Zarafshan,et al.  A New Stabilization Algorithm for a Two-Wheeled Mobile Robot Aided by Reaction Wheel , 2015 .

[50]  Vijay R. Konda,et al.  Actor-Critic Algorithms , 1999, NIPS.