Deep multi-level fusion network for multi-source image pixel-wise classification

Abstract For multi-source image pixel-wise classification, each image information is different and complementary in the same area or scene. However, how to integrate them for decision-making is a difficult problem. In this paper, we focus on the characteristics of multi-source image and propose a novel pixel-wise classification method, named deep multi-level fusion network. The proposed method is to classify multi-sensor data including very high-resolution (VHR) RGB imagery, hyperspectral imagery (HSI) and multispectral light detection and ranging (MS-LiDAR) point cloud data. First, a deep spectral–spatial attention network is proposed to process HSI and MS-LiDAR images and get a learned classification map, which is based on feature level fusion. Next, a down-superpixel segmentation algorithm is proposed to get a segmentation result for VHR RGB imagery. Finally, the feature level fusion results are refinement by the down-superpixel segmentation results on the decision level, and get the final result. Extensive experiments and analyses on the data set g r s s _ d f c _ 2018 demonstrate that the proposed multi-level fusion network can achieve a better result in the multi-source image pixel-wise classification.

[1]  Walid Ouerghemmi,et al.  Decision Fusion of Remote-Sensing Data for Land Cover Classification , 2019, Multimodal Scene Understanding.

[2]  Ben Somers,et al.  Enhancing the performance of Multiple Endmember Spectral Mixture Analysis (MESMA) for urban land cover mapping using airborne lidar data and band selection , 2019, Remote Sensing of Environment.

[3]  Shuyuan Yang,et al.  A Dual-Branch Attention fusion deep network for multiresolution remote-Sensing image classification , 2020, Inf. Fusion.

[4]  Xiao Xiang Zhu,et al.  Hyperspectral and LiDAR Data Fusion Using Extinction Profiles and Deep Convolutional Neural Network , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[5]  Bin Hu,et al.  Feature-level fusion approaches based on multimodal EEG data for depression recognition , 2020, Inf. Fusion.

[6]  Pedram Ghamisi,et al.  Fusion of Hyperspectral and LiDAR Data Using Sparse and Low-Rank Component Analysis , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[7]  Xiaohui Huang,et al.  A feature selection approach for hyperspectral image based on modified ant lion optimizer , 2019, Knowl. Based Syst..

[8]  Yi Yang,et al.  Gated Channel Transformation for Visual Recognition , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Hui Zhou,et al.  Multi-branch fusion network for hyperspectral image classification , 2019, Knowl. Based Syst..

[10]  Licheng Jiao,et al.  Dense connection and depthwise separable convolution based CNN for polarimetric SAR image classification , 2020, Knowl. Based Syst..

[11]  Chenglin Wen,et al.  Multiscale fused network with additive channel-spatial attention for image segmentation , 2021, Knowl. Based Syst..

[12]  Zhenyu He,et al.  Hierarchical spatial-aware Siamese network for thermal infrared object tracking , 2017, Knowl. Based Syst..

[13]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[14]  Deyu Meng,et al.  Multi-scale generative adversarial inpainting network based on cross-layer attention transfer mechanism , 2020, Knowl. Based Syst..

[15]  Michele Volpi,et al.  Deep multi-task learning for a geographically-regularized semantic segmentation of aerial images , 2018, ISPRS Journal of Photogrammetry and Remote Sensing.

[16]  In-So Kweon,et al.  CBAM: Convolutional Block Attention Module , 2018, ECCV.

[17]  Ryan A. Rossi,et al.  Attention Models in Graphs , 2018, ACM Trans. Knowl. Discov. Data.

[18]  Naoto Yokoya,et al.  2018 IEEE GRSS Data Fusion Contest: Multimodal Land Use Classification [Technical Committees] , 2018 .

[19]  Bing Liu,et al.  Remote sensing image captioning via Variational Autoencoder and Reinforcement Learning , 2020, Knowl. Based Syst..

[20]  Fang Liu,et al.  Task-Oriented GAN for PolSAR Image Classification and Clustering , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[21]  Jocelyn Chanussot,et al.  Braids of partitions for the hierarchical representation and segmentation of multimodal images , 2019, Pattern Recognit..

[22]  Yongfeng Huang,et al.  Multi-source data fusion for aspect-level sentiment classification , 2020, Knowl. Based Syst..

[23]  Chuanjun Zhao,et al.  Multi-source domain adaptation with joint learning for cross-domain sentiment classification , 2020, Knowl. Based Syst..

[24]  Xiaogang Wang,et al.  Residual Attention Network for Image Classification , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Somnath Dey,et al.  A novel hybrid score level and decision level fusion scheme for cancelable multi-biometric verification , 2018, Applied Intelligence.

[26]  Luciano Alparone,et al.  Sensitivity of Pansharpening Methods to Temporal and Instrumental Changes Between Multispectral and Panchromatic Data Sets , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[27]  Tao Xiang,et al.  Leader-Based Multi-Scale Attention Deep Architecture for Person Re-Identification , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Shaowen Wang,et al.  A 3D convolutional neural network method for land cover classification using LiDAR and multi-temporal Landsat imagery , 2018, ISPRS Journal of Photogrammetry and Remote Sensing.

[29]  G. Boynton,et al.  Global effects of feature-based attention in human visual cortex , 2002, Nature Neuroscience.

[30]  Yao Zhao,et al.  EA-LSTM: Evolutionary Attention-based LSTM for Time Series Prediction , 2018, Knowl. Based Syst..

[31]  Pascal Fua,et al.  SLIC Superpixels Compared to State-of-the-Art Superpixel Methods , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Xianbiao Qi,et al.  Fast infrared and visible image fusion with structural decomposition , 2020, Knowl. Based Syst..

[33]  Sen Jia,et al.  Multiple Feature-Based Superpixel-Level Decision Fusion for Hyperspectral and LiDAR Data Classification , 2021, IEEE Transactions on Geoscience and Remote Sensing.

[34]  Adrian Hilton,et al.  Channel and spatial attention based deep object co-segmentation , 2021, Knowl. Based Syst..

[35]  Priti P. Rege,et al.  Pixel level fusion techniques for SAR and optical images: A review , 2020, Inf. Fusion.

[36]  Jenq-Neng Hwang,et al.  Effective person re-identification by self-attention model guided feature learning , 2020, Knowl. Based Syst..

[37]  Jin Zhao,et al.  Deep Multiple Instance Learning-Based Spatial–Spectral Classification for PAN and MS Imagery , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[38]  Timothy C. Havens,et al.  Efficient Multiple Kernel Classification Using Feature and Decision Level Fusion , 2017, IEEE Transactions on Fuzzy Systems.

[39]  Yann Gousseau,et al.  LSDSAR, a Markovian a contrario framework for line segment detection in SAR images , 2020, Pattern Recognit..

[40]  Junwei Han,et al.  Learning Compact and Discriminative Stacked Autoencoder for Hyperspectral Image Classification , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[41]  Qian Du,et al.  Multisource Remote Sensing Data Classification Based on Convolutional Neural Network , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[42]  Xu Liu,et al.  Polarimetric Convolutional Network for PolSAR Image Classification , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[43]  Luciano Alparone,et al.  Spatial Methods for Multispectral Pansharpening: Multiresolution Analysis Demystified , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[44]  Tom Duckett,et al.  Online learning for 3D LiDAR-based human detection: experimental analysis of point cloud clustering and classification methods , 2019, Autonomous Robots.

[45]  Jiewen Zhao,et al.  Intelligent evaluation of total volatile basic nitrogen (TVB-N) content in chicken meat by an improved multiple level data fusion model , 2017 .

[46]  Fangzhao Wu,et al.  Domain attention model for multi-domain sentiment classification , 2018, Knowl. Based Syst..

[47]  Jun Zhou,et al.  Inverse Coefficient of Variation Feature and Multilevel Fusion Technique for Hyperspectral and LiDAR Data Classification , 2020, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[48]  Omar Nasr,et al.  RGB and LiDAR fusion based 3D Semantic Segmentation for Autonomous Driving , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).

[49]  Chen Shen,et al.  Hyperspectral image classification based on discriminative locality preserving broad learning system , 2020, Knowl. Based Syst..

[50]  Xuelong Li,et al.  Stacked Fisher autoencoder for SAR change detection , 2019, Pattern Recognit..