Reinforcement learning control for underactuated surface vessel with output error constraints and uncertainties

Abstract This study investigates the trajectory tracking control problem of an underactuated marine vessel in the presence of output constraints, model uncertainties and environmental disturbances. The error transformation technique can ensure that the tracking errors remain within the predefined constraint boundaries. The controller is designed in combination with the critic function and the reinforcement learning (RL) algorithm based on actor-critic neural networks. The RL method is applied to solve model uncertainties and disturbances, and the critic function modifies the control action to supervise the system performance. Based on Lyapunov’s direct method, a stability analysis is proposed to prove that the boundedness of system signals and the desired tracking performance can be guaranteed. Finally, the simulation illustrates the effectiveness and feasibility of the proposed controller.

[1]  Xianku Zhang,et al.  Practical Robust Neural Path Following Control for Underactuated Marine Vessels with Actuators Uncertainties , 2017 .

[2]  Ji-Hong Li.  Path tracking of underactuated ships with general form of dynamics , 2016, Int. J. Control.

[3]  Sen Wang,et al.  Adaptive low-level control of autonomous underwater vehicles using deep reinforcement learning , 2018, Robotics Auton. Syst..

[4]  Chunsheng Liu,et al.  Disturbance observer-based robust missile autopilot design with full-state constraints via adaptive dynamic programming , 2018, J. Frankl. Inst..

[5]  Changyin Sun,et al.  Control Design of a Marine Vessel System Using Reinforcement Learning , 2018, Neurocomputing.

[6]  Chuan-Kai Lin,et al.  Hinfinity reinforcement learning control of robot manipulators using fuzzy wavelet networks , 2009, Fuzzy Sets Syst..

[7]  Jae Young Lee,et al.  Integral Reinforcement Learning for Continuous-Time Input-Affine Nonlinear Systems With Simultaneous Invariant Explorations , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[8]  Yong-Kon Lim,et al.  Point-to-point navigation of underactuated ships , 2008, Autom..

[9]  Zewei Zheng,et al.  Path Following of a Surface Vessel With Prescribed Performance in the Presence of Input Saturation and External Disturbances , 2017, IEEE/ASME Transactions on Mechatronics.

[10]  Derong Liu,et al.  Neural-Network-Based Online HJB Solution for Optimal Robust Guaranteed Cost Control of Continuous-Time Uncertain Nonlinear Systems , 2014, IEEE Transactions on Cybernetics.

[11]  Khac Duc Do,et al.  Path-tracking control of underactuated ships under tracking error constraints , 2015 .

[12]  Yueying Wang,et al.  Fuzzy Uncertainty Observer-Based Path-Following Control of Underactuated Marine Vehicles with Unmodeled Dynamics and Disturbances , 2018, 2017 International Conference on Fuzzy Theory and Its Applications (iFUZZY).

[13]  Chenglong Wang,et al.  Adaptive Neural-Based Finite-Time Trajectory Tracking Control for Underactuated Marine Surface Vessels With Position Error Constraint , 2019, IEEE Access.

[14]  Weidong Zhang,et al.  Robust adaptive trajectory tracking control of underactuated surface vessel in fields of marine practice , 2018 .

[15]  Lu Liu,et al.  Path following of marine surface vehicles with dynamical uncertainty and time-varying ocean disturbances , 2016, Neurocomputing.

[16]  Zhixiang Liu,et al.  Unmanned surface vehicles: An overview of developments and challenges , 2016, Annu. Rev. Control..

[17]  Shuzhi Sam Ge,et al.  Optimized Backstepping for Tracking Control of Strict-Feedback Systems , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[18]  Qiuye Sun,et al.  Adaptive critic design-based robust neural network control for nonlinear distributed parameter systems with unknown dynamics , 2015, Neurocomputing.

[19]  Liang Sun,et al.  Path following control for marine surface vessel with uncertainties and input saturation , 2016, Neurocomputing.

[20]  Robert Babuska,et al.  Actor-critic reinforcement learning for tracking control in robotics , 2016, 2016 IEEE 55th Conference on Decision and Control (CDC).

[21]  Weisheng Yan,et al.  Integral Reinforcement Learning-Based Adaptive NN Control for Continuous-Time Nonlinear MIMO Systems With Unknown Control Directions , 2020, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[22]  Hao Wang,et al.  Predictor-based LOS guidance law for path following of underactuated marine surface vehicles with sideslip compensation , 2016 .

[23]  ZhangGuoqing,et al.  Practical Robust Neural Path Following Control for Underactuated Marine Vessels with Actuators Uncertainties , 2017 .

[24]  Dan Wang,et al.  Adaptive Dynamic Surface Control for Formations of Autonomous Surface Vehicles With Uncertain Dynamics , 2013, IEEE Transactions on Control Systems Technology.

[25]  Lihua Xie,et al.  Adaptive Trajectory Tracking Control of a Fully Actuated Surface Vessel With Asymmetrically Constrained Input and Output , 2018, IEEE Transactions on Control Systems Technology.

[26]  Zhongjiu Zheng,et al.  Finite-Time Sideslip Observer-Based Adaptive Fuzzy Path-Following Control of Underactuated Marine Vehicles with Time-Varying Large Sideslip , 2018, Int. J. Fuzzy Syst..

[27]  Xu Jin,et al.  Nonrepetitive Leader–Follower Formation Tracking for Multiagent Systems With LOS Range and Angle Constraints Using Iterative Learning Control , 2019, IEEE Transactions on Cybernetics.

[28]  Lihua Xie,et al.  Error-Constrained LOS Path Following of a Surface Vessel With Actuator Saturation and Faults , 2018, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[29]  Lihua Xie,et al.  Finite-time path following control for a stratospheric airship with input saturation and error constraint , 2019, Int. J. Control.

[30]  Kristin Y. Pettersen,et al.  Collision Avoidance for Underactuated Marine Vehicles Using the Constant Avoidance Angle Algorithm , 2020, IEEE Transactions on Control Systems Technology.

[31]  Xu Jin,et al.  Fault-tolerant iterative learning control for mobile robots non-repetitive trajectory tracking with output constraints , 2018, Autom..

[32]  Guang-Hong Yang,et al.  Quantization-Based Adaptive Actor-Critic Tracking Control With Tracking Error Constraints , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[33]  Yang Li,et al.  Adaptive Neural Network Control of AUVs With Control Input Nonlinearities Using Reinforcement Learning , 2017, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[34]  Khac Duc Do,et al.  Global robust adaptive path-tracking control of underactuated ships under stochastic disturbances , 2016 .

[35]  Ming Zhu,et al.  Trajectory tracking control for a marine surface vessel with asymmetric saturation actuators , 2017, Robotics Auton. Syst..

[36]  Ding Wang,et al.  Adaptive-Critic-Based Robust Trajectory Tracking of Uncertain Dynamics and Its Application to a Spring–Mass–Damper System , 2018, IEEE Transactions on Industrial Electronics.

[37]  Xu Jin,et al.  Fault Tolerant Nonrepetitive Trajectory Tracking for MIMO Output Constrained Nonlinear Systems Using Iterative Learning Control , 2019, IEEE Transactions on Cybernetics.

[38]  Khac Duc Do,et al.  Practical control of underactuated ships , 2010 .

[39]  Meng Joo Er,et al.  Direct Adaptive Fuzzy Tracking Control of Marine Vehicles With Fully Unknown Parametric Dynamics and Uncertainties , 2016, IEEE Transactions on Control Systems Technology.

[40]  Ming Zhu,et al.  Output-constrained tracking control of an underactuated autonomous underwater vehicle with uncertainties , 2019, Ocean Engineering.

[41]  João P. Hespanha,et al.  Trajectory-Tracking and Path-Following of Underactuated Autonomous Vehicles With Parametric Modeling Uncertainty , 2007, IEEE Transactions on Automatic Control.

[42]  Lu Liu,et al.  Neural adaptive steering of an unmanned surface vehicle with measurement noises , 2016, Neurocomputing.

[43]  Shaocheng Tong,et al.  Adaptive neural control using reinforcement learning for a class of robot manipulator , 2014, Neural Computing and Applications.

[44]  Tristan Perez,et al.  Trajectory tracking passivity-based control for marine vehicles subject to disturbances , 2017, J. Frankl. Inst..

[45]  Chenglong Wang,et al.  High-Gain Observer-Based Line-of-Sight Guidance for Adaptive Neural Path Following Control of Underactuated Marine Surface Vessels , 2019, IEEE Access.

[46]  H. Wang,et al.  Adaptive neural control for cooperative path following of marine surface vehicles: state and output feedback , 2016, Int. J. Syst. Sci..

[47]  Shuzhi Sam Ge,et al.  Adaptive Neural Network Control of a Fully Actuated Marine Surface Vessel With Multiple Output Constraints , 2014, IEEE Transactions on Control Systems Technology.

[48]  Lionel Lapierre,et al.  Survey on Fuzzy-Logic-Based Guidance and Control of Marine Surface Vehicles and Underwater Vehicles , 2018, Int. J. Fuzzy Syst..

[49]  Youxian Sun,et al.  Robust ADP Design for Continuous-Time Nonlinear Systems With Output Constraints , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[50]  Khoshnam Shojaei,et al.  Neural adaptive robust control of underactuated marine surface vehicles with input saturation , 2015 .

[51]  Guang-Hong Yang,et al.  Adaptive Actor–Critic Design-Based Integral Sliding-Mode Control for Partially Unknown Nonlinear Systems With Input Disturbances , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[52]  Kristin Ytterstad Pettersen,et al.  Observer Based Path Following for Underactuated Marine Vessels in the Presence of Ocean Currents: A Global Approach - With proofs , 2018, ArXiv.