Ship Classification Based on Attention Mechanism and Multi-Scale Convolutional Neural Network for Visible and Infrared Images

Visible image quality is very susceptible to changes in illumination, and there are limitations in ship classification using images acquired by a single sensor. This study proposes a ship classification method based on an attention mechanism and multi-scale convolutional neural network (MSCNN) for visible and infrared images. First, the features of visible and infrared images are extracted by a two-stream symmetric multi-scale convolutional neural network module, and then concatenated to make full use of the complementary features present in multi-modal images. After that, the attention mechanism is applied to the concatenated fusion features to emphasize local details areas in the feature map, aiming to further improve feature representation capability of the model. Lastly, attention weights and the original concatenated fusion features are added element by element and fed into fully connected layers and Softmax output layer for final classification output. Effectiveness of the proposed method is verified on a visible and infrared spectra (VAIS) dataset, which shows 93.81% accuracy in classification results. Compared with other state-of-the-art methods, the proposed method could extract features more effectively and has better overall classification performance.

[1]  John Reeder,et al.  Convolution neural networks for ship type recognition , 2016, SPIE Defense + Security.

[2]  Huarong Jia,et al.  Marine ship recognition based on cascade CNNs , 2020, Target Recognition and Artificial Intelligence Summit Forum.

[3]  Zhao Baojun,et al.  Ship classification based on convolutional neural networks , 2019, The Journal of Engineering.

[4]  Hongwei Liu,et al.  Convolutional Neural Network With Data Augmentation for SAR Target Recognition , 2016, IEEE Geoscience and Remote Sensing Letters.

[5]  Qing Fei,et al.  Convolutional Neural Network Based on Extreme Learning Machine for Maritime Ships Recognition in Infrared Images , 2018, Sensors.

[6]  Michael T. Wolf,et al.  VAIS: A dataset for recognizing maritime imagery in the visible and infrared spectrums , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[7]  Gholamreza Akbarizadeh,et al.  Ship Classification in SAR Images Using a New Hybrid CNN–MLP Classifier , 2018, Journal of the Indian Society of Remote Sensing.

[8]  Krištof Oštir,et al.  Vessel detection and classification from spaceborne optical images: A literature survey , 2018, Remote sensing of environment.

[9]  Qian Du,et al.  Local Binary Patterns and Extreme Learning Machine for Hyperspectral Imagery Classification , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[10]  Fan Yang,et al.  Multi-feature fusion of convolutional neural networks for Fine-Grained ship classification , 2019, J. Intell. Fuzzy Syst..

[11]  Zyad Shaaban,et al.  Data Mining: A Preprocessing Engine , 2006 .

[12]  刘峰 Liu Feng,et al.  Ship recognition based on multi-band deep neural network , 2017 .

[13]  Shengxiang Qi,et al.  Ship detection based on rotation-invariant HOG descriptors for airborne infrared images , 2018, International Symposium on Multispectral Image Processing and Pattern Recognition.

[14]  Jin Wang,et al.  Lightweight deep network for traffic sign classification , 2019, Annals of Telecommunications.

[15]  Xiaolei Zhao,et al.  Residual Dense Network Based on Channel-Spatial Attention for the Scene Classification of a High-Resolution Remote Sensing Image , 2020, Remote. Sens..

[16]  Frédéric Bouchara,et al.  Multimodal Deep Learning for Robust Recognizing Maritime Imagery in the Visible and Infrared Spectrums , 2018, ICIAR.

[17]  Léon Bottou,et al.  Large-Scale Machine Learning with Stochastic Gradient Descent , 2010, COMPSTAT.

[18]  Jinjun Tang,et al.  Ship Type Recognition via a Coarse-to-Fine Cascaded Convolution Neural Network , 2020, Journal of Navigation.

[19]  Zongliang Gan,et al.  Residual Group Channel and Space Attention Network for Hyperspectral Image Classification , 2020, Remote. Sens..

[20]  Gang Sun,et al.  Squeeze-and-Excitation Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[21]  In-So Kweon,et al.  CBAM: Convolutional Block Attention Module , 2018, ECCV.

[22]  沈同圣 Shen Tongsheng,et al.  Convolutional Neural Network Based Multi-Band Ship Target Recognition with Feature Fusion , 2017 .

[23]  Lianru Gao,et al.  Deep CNN With Multi-Scale Rotation Invariance Features for Ship Classification , 2018, IEEE Access.

[24]  Shibin Parameswaran,et al.  Vessel classification in overhead satellite imagery using weighted "bag of visual words" , 2015, Defense + Security Symposium.

[25]  Se-Young Oh,et al.  Fast training of convolutional neural network classifiers through extreme learning machines , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[26]  Guangfeng Lin,et al.  Classification of Marine Vessels with Multi-Feature Structure Fusion , 2019, Applied Sciences.

[27]  Dong Wang,et al.  Ship Target Detection Algorithm Based on Improved Faster R-CNN , 2019, Electronics.

[28]  Jungong Han,et al.  Pruning Convolutional Neural Networks with an Attention Mechanism for Remote Sensing Image Classification , 2020, Electronics.

[29]  Dipankar Das,et al.  Enhanced SenticNet with Affective Labels for Concept-Based Opinion Mining , 2013, IEEE Intelligent Systems.