Video-Based Detection Infrastructure Enhancement for Automated Ship Recognition and Behavior Analysis

Video-based detection infrastructure is crucial for promoting connected and autonomous shipping (CAS) development, which provides critical on-site traffic data for maritime participants. Ship behavior analysis, one of the fundamental tasks for fulfilling smart video-based detection infrastructure, has become an active topic in the CAS community. Previous studies focused on ship behavior analysis by exploring spatial-temporal information from automatic identification system (AIS) data, and less attention was paid to maritime surveillance videos. To bridge the gap, we proposed an ensemble you only look once (YOLO) framework for ship behavior analysis. First, we employed the convolutional neural network in the YOLO model to extract multi-scaled ship features from the input ship images. Second, the proposed framework generated many bounding boxes (i.e., potential ship positions) based on the object confidence level. Third, we suppressed the background bounding box interferences, and determined ship detection results with intersection over union (IOU) criterion, and thus obtained ship positions in each ship image. Fourth, we analyzed spatial-temporal ship behavior in consecutive maritime images based on kinematic ship information. The experimental results have shown that ships are accurately detected (i.e., both of the average recall and precision rate were higher than 90%) and the historical ship behaviors are successfully recognized. The proposed framework can be adaptively deployed in the connected and autonomous vehicle detection system in the automated terminal for the purpose of exploring the coupled interactions between traffic flow variation and heterogeneous detection infrastructures, and thus enhance terminal traffic network capacity and safety.

[1]  Deepu Rajan,et al.  Video Processing From Electro-Optical Sensors for Object Detection and Tracking in a Maritime Environment: A Survey , 2016, IEEE Transactions on Intelligent Transportation Systems.

[2]  Jinjun Tang,et al.  Anomaly Detection and Cleaning of Highway Elevation Data from Google Earth Using Ensemble Empirical Mode Decomposition , 2018 .

[3]  Alexander Wong,et al.  Fast YOLO: A Fast You Only Look Once System for Real-time Embedded Object Detection in Video , 2017, ArXiv.

[4]  Bin Wang,et al.  Detecting Infrared Maritime Targets Overwhelmed in Sun Glitters by Antijitter Spatiotemporal Saliency , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[5]  Bo Du,et al.  Deep Learning for Remote Sensing Data: A Technical Tutorial on the State of the Art , 2016, IEEE Geoscience and Remote Sensing Magazine.

[6]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[7]  Hyuk-Jae Lee,et al.  A High-Throughput and Power-Efficient FPGA Implementation of YOLO CNN for Object Detection , 2019, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[8]  Zhenwei Shi,et al.  Fully Convolutional Network With Task Partitioning for Inshore Ship Detection in Optical Remote Sensing Images , 2017, IEEE Geoscience and Remote Sensing Letters.

[9]  K. Saravanan,et al.  How to prevent maritime border collision for fisheries?-A design of Real-Time Automatic Identification System , 2018, Earth Science Informatics.

[10]  Giuliana Pallotta,et al.  Maritime Traffic Networks: From Historical Positioning Data to Unsupervised Maritime Traffic Monitoring , 2018, IEEE Transactions on Intelligent Transportation Systems.

[11]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Zhao Lin,et al.  Contextual Region-Based Convolutional Neural Network with Multilayer Fusion for SAR Ship Detection , 2017, Remote. Sens..

[13]  Yang Zhang,et al.  Ship detection for visual maritime surveillance from non-stationary platforms☆ , 2017 .

[14]  Raghu G. Raj,et al.  Practical Multichannel SAR Imaging in the Maritime Environment , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[15]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Thia Kirubarajan,et al.  Measurement level AIS/radar fusion , 2015, Signal Process..

[17]  Mohsen Guizani,et al.  Self-Selective Correlation Ship Tracking Method for Smart Ocean System , 2019, ArXiv.

[18]  Rolf J. Bye,et al.  Maritime navigation accidents and risk indicators: An exploratory statistical analysis using AIS data and accident reports , 2018, Reliab. Eng. Syst. Saf..

[19]  Floris Goerlandt,et al.  An Advanced Method For Detecting Possible Near Miss Ship Collisions From AIS Data , 2016 .

[20]  Huafeng Wu,et al.  Robust Ship Tracking via Multi-view Learning and Sparse Representation , 2018, Journal of Navigation.

[21]  Weiming Zhang,et al.  Vessel Spatio-temporal Knowledge Discovery with AIS Trajectories Using Co-clustering , 2017 .

[22]  Shuang Li,et al.  Online Prediction of Ship Behavior with Automatic Identification System Sensor Data Using Bidirectional Long Short-Term Memory Recurrent Neural Network , 2018, Sensors.

[23]  Guojun Gan,et al.  K-means Clustering with Outlier Removal , 2017, Pattern Recognit. Lett..

[24]  Fabio Mazzarella,et al.  SAR Ship Detection and Self-Reporting Data Fusion Based on Traffic Knowledge , 2015, IEEE Geoscience and Remote Sensing Letters.

[25]  Jakub Montewka,et al.  A method for detecting possible near miss ship collisions from AIS data , 2015 .

[26]  Nakwan Kim,et al.  Dynamic model identification of unmanned surface vehicles using deep learning network , 2018, Applied Ocean Research.

[27]  Naixue Xiong,et al.  Spatio-Temporal Vessel Trajectory Clustering Based on Data Mapping and Density , 2018, IEEE Access.

[28]  Konstantinos Tserpes,et al.  Employing traditional machine learning algorithms for big data streams analysis: The case of object trajectory prediction , 2016, J. Syst. Softw..

[29]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[30]  Liangbin Zhao,et al.  A trajectory clustering method based on Douglas-Peucker compression and density for marine traffic pattern recognition , 2019 .

[31]  Zhimin Zhou,et al.  GF-4 Satellite and Automatic Identification System Data Fusion for Ship Tracking , 2019, IEEE Geoscience and Remote Sensing Letters.