An Autonomous Path Planning Model for Unmanned Ships Based on Deep Reinforcement Learning

Deep reinforcement learning (DRL) has excellent performance in continuous control problems and it is widely used in path planning and other fields. An autonomous path planning model based on DRL is proposed to realize the intelligent path planning of unmanned ships in the unknown environment. The model utilizes the deep deterministic policy gradient (DDPG) algorithm, through the continuous interaction with the environment and the use of historical experience data; the agent learns the optimal action strategy in a simulation environment. The navigation rules and the ship’s encounter situation are transformed into a navigation restricted area, so as to achieve the purpose of planned path safety in order to ensure the validity and accuracy of the model. Ship data provided by ship automatic identification system (AIS) are used to train this path planning model. Subsequently, the improved DRL is obtained by combining DDPG with the artificial potential field. Finally, the path planning model is integrated into the electronic chart platform for experiments. Through the establishment of comparative experiments, the results show that the improved model can achieve autonomous path planning, and it has good convergence speed and stability.

[1]  Paul Rad,et al.  Driverless vehicle security: Challenges and future research opportunities , 2020, Future Gener. Comput. Syst..

[2]  Meixin Zhu,et al.  Human-Like Autonomous Car-Following Model with Deep Reinforcement Learning , 2018, Transportation Research Part C: Emerging Technologies.

[3]  David Isele,et al.  Navigating Occluded Intersections with Autonomous Vehicles Using Deep Reinforcement Learning , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[4]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[5]  Ming Liu,et al.  Virtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[6]  Joelle Pineau,et al.  An Actor-Critic Algorithm for Sequence Prediction , 2016, ICLR.

[7]  Frédéric Plumet,et al.  Reactive path planning for autonomous sailboat , 2011, 2011 15th International Conference on Advanced Robotics (ICAR).

[8]  Yang Li,et al.  A Ship Rotation Detection Model in Remote Sensing Images Based on Feature Fusion Pyramid Network and Deep Reinforcement Learning , 2018, Remote. Sens..

[9]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[10]  Ying Cui,et al.  An Improved Genetic Algorithm for Path-Planning of Unmanned Surface Vehicle , 2019, Sensors.

[11]  Zhixiang Liu,et al.  Unmanned surface vehicles: An overview of developments and challenges , 2016, Annu. Rev. Control..

[12]  Rubo Zhang,et al.  An adaptive obstacle avoidance algorithm for unmanned surface vehicle in complicated marine environments , 2014, IEEE/CAA Journal of Automatica Sinica.

[13]  Youyong Kong,et al.  Deep Direct Reinforcement Learning for Financial Signal Representation and Trading , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[14]  Lingli Yu,et al.  Intelligent Land-Vehicle Model Transfer Trajectory Planning Method Based on Deep Reinforcement Learning , 2018, Sensors.

[15]  David Clelland,et al.  Automatic simulation of ship navigation , 2011 .

[16]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[17]  Robert Sutton,et al.  Intelligent ship autopilots––A historical perspective , 2003 .

[18]  Chen Guo,et al.  Automatic collision avoidance of multiple ships based on deep Q-learning , 2019, Applied Ocean Research.

[19]  Michael T. Wolf,et al.  Safe Maritime Autonomous Navigation With COLREGS, Using Velocity Obstacles , 2014, IEEE Journal of Oceanic Engineering.

[20]  Xiang Chen,et al.  Decision-Making for the Autonomous Navigation of Maritime Autonomous Surface Ships Based on Scene Division and Deep Reinforcement Learning , 2019, Sensors.

[21]  Bai Ting-ying.  International Standards of Electronic Chart Display and Information System , 2004 .

[22]  C Soares,et al.  Multi-objective evolutionary algorithm in ship route optimization , 2014 .

[23]  Will Serrano,et al.  Deep Reinforcement Learning Algorithms in Intelligent Infrastructure , 2019, Infrastructures.

[24]  S Mankabady.  THE INTERNATIONAL MARITIME ORGANIZATION, VOLUME 1: INTERNATIONAL SHIPPING RULES , 1986 .

[25]  A. Lazarowska.  Ship's Trajectory Planning for Collision Avoidance at Sea Based on Ant Colony Optimisation , 2015 .

[26]  Lokukaluge P. Perera,et al.  Autonomous guidance and navigation based on the COLREGs rules and regulations of collision avoidance. , 2010 .

[27]  Chen Chen,et al.  A knowledge-free path planning approach for smart ships based on reinforcement learning , 2019, Ocean Engineering.

[28]  Wasif Naeem,et al.  A Rule-based Heuristic Method for COLREGS-compliant Collision Avoidance for an Unmanned Surface Vehicle , 2012 .

[29]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[30]  Yan Wang,et al.  Unmanned Surface Vehicle Course Tracking Control Based on Neural Network and Deep Deterministic Policy Gradient Algorithm , 2018, 2018 OCEANS - MTS/IEEE Kobe Techno-Oceans (OTO).

[31]  Xiumin Chu,et al.  Ship predictive collision avoidance method based on an improved beetle antennae search algorithm , 2019, Ocean Engineering.

[32]  Taolue Chen,et al.  Road Tracking Using Deep Reinforcement Learning for Self-driving Car Applications , 2019, CORES.

[33]  Wu Zhao-lin.  Risk analysis system of underway ships in heavy sea , 2004 .

[34]  Qingnian Zhang,et al.  Research on Autonomous Navigation Control of Unmanned Ship Based on Unity3D , 2019, 2019 5th International Conference on Control, Automation and Robotics (ICCAR).

[35]  K. Simmonds,et al.  The International Maritime Organization , 1994 .

[36]  Sun,et al.  Research into the Automatic Berthing of Underactuated Unmanned Ships under Wind Loads Based on Experiment and Numerical Analysis , 2019, Journal of Marine Science and Engineering.

[37]  Mostefa Mohamed-Seghir,et al.  Comparison of Computational Intelligence Methods Based on Fuzzy Sets and Game Theory in the Synthesis of Safe Ship Control Based on Information from a Radar ARPA System , 2019, Remote. Sens..

[38]  Myung-Il Roh,et al.  COLREGs-compliant multiship collision avoidance based on deep reinforcement learning , 2019, Ocean Engineering.

[39]  Eduardo F. Morales,et al.  An Introduction to Reinforcement Learning , 2011 .