Trajectory tracking algorithm for autonomous vehicles using adaptive reinforcement learning

The off-shore industry requires periodic monitoring of underwater infrastructure for preventive maintenance and security reasons. The tasks in hostile environments can be achieved automatically through autonomous robots like UUV, AUV and ASV. When the robot controllers are based on prespecified conditions they could not function properly in these hostile changing environments. It is beneficial to count with adaptive control strategies that are capable to readapt the control policies when deviations, from the desired behavior, are detected. In this paper, we present an online selective reinforcement learning approach for learning reference tracking control policies given no prior knowledge of the dynamical system. The proposed approach enables real-time adaptation of the control policies, executed by the adaptive controller, based on ongoing interactions with the non-stationary environments. Applications on surface vehicles under nonstationary and perturbed environments are simulated. The presented simulation results demonstrate the performance of this novel algorithm to find optimal control policies that solve the trajectory tracking control task in unmanned vehicles.

[1]  Stephen P. Boyd,et al.  ROBUST LINEAR PROGRAMMING AND OPTIMAL CONTROL , 2002 .

[2]  Carl E. Rasmussen,et al.  Gaussian process dynamic programming , 2009, Neurocomputing.

[3]  Bernhard Schölkopf,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2005, IEEE Transactions on Neural Networks.

[4]  Jan Peters,et al.  Reinforcement learning in robotics: A survey , 2013, Int. J. Robotics Res..

[5]  M. Sreekumar,et al.  Tracking Algorithm Using Leader Follower Approach for Multi Robots , 2013 .

[6]  Lucas C. McNinch,et al.  Review of nonlinear tracking and setpoint control approaches for autonomous underactuated marine vehicles , 2010, Proceedings of the 2010 American Control Conference.

[7]  Wettergreen Alexander Zelinsky.  Reinforcement Learning applied to the control of an AutonolDOlls ·U nclerwater .Vehicle , 1998 .

[8]  Shie Mannor,et al.  The kernel recursive least-squares algorithm , 2004, IEEE Transactions on Signal Processing.

[9]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[10]  A. Atiya,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2005, IEEE Transactions on Neural Networks.

[11]  Peter Stone,et al.  RTMBA: A Real-Time Model-Based Reinforcement Learning Architecture for robot control , 2011, 2012 IEEE International Conference on Robotics and Automation.

[12]  Pierre-Yves Oudeyer,et al.  Active learning of inverse models with intrinsically motivated goal exploration in robots , 2013, Robotics Auton. Syst..

[13]  Alexander Zelinsky,et al.  Reinforcement Learning applied to the control of an Autonomous Underwater Vehicle , 1999 .

[14]  Kagan Tumer,et al.  Adaptive navigation for autonomous robots , 2011, Robotics Auton. Syst..

[15]  Jan Peters,et al.  Incremental online sparsification for model learning in real-time robot control , 2011, Neurocomputing.

[16]  Olivier Sigaud,et al.  On-line regression algorithms for learning mechanical models of robots: A survey , 2011, Robotics Auton. Syst..

[17]  Feng Lin,et al.  An optimal control approach to robust tracking of linear systems , 2009, Int. J. Control.

[18]  Marc Peter Deisenroth,et al.  Efficient reinforcement learning using Gaussian processes , 2010 .

[19]  Mohammad A. Jaradat,et al.  Reinforcement based mobile robot navigation in dynamic environment , 2011 .

[20]  George M. Siouris,et al.  Applied Optimal Control: Optimization, Estimation, and Control , 1979, IEEE Transactions on Systems, Man, and Cybernetics.

[21]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[22]  Thor I. Fossen,et al.  Marine Control Systems Guidance, Navigation, and Control of Ships, Rigs and Underwater Vehicles , 2002 .

[23]  Gregor Klan.  Tracking-error model-based predictive control for mobile robots in real time , 2007 .

[24]  Gunnar Rätsch,et al.  Input space versus feature space in kernel-based methods , 1999, IEEE Trans. Neural Networks.

[25]  J. Batlle,et al.  A behavior-based scheme using reinforcement learning for autonomous underwater vehicles , 2005, IEEE Journal of Oceanic Engineering.