ABCNet: Attentive bilateral contextual network for efficient semantic segmentation of Fine-Resolution remotely sensed imagery ISPRS Journal of Photogrammetry and Remote Sensing

computational demand. Particularly, the frequently-used fully convolutional network (FCN) relies heavily on fine-grained spatial detail (fine spatial resolution) and contextual information (large receptive fields), both imposing high computational costs. This impedes the practical utility of FCN for real-world applications, especially those requiring real-time data processing. In this paper, we propose a novel Attentive Bilateral Contextual Network (ABCNet), a lightweight convolutional neural network (CNN) with a spatial path and a contextual path. Extensive experiments, including a comprehensive ablation study, demonstrate that ABCNet has strong discrimination capability with competitive accuracy compared with state- of-the-art benchmark methods while achieving significantly increased computational efficiency. Specifically, the proposed ABCNet achieves a 91.3% overall accuracy (OA) on the Potsdam test dataset and outperforms all lightweight benchmark methods significantly. The code is freely available at https://github. com/lironui/ABCNet.

[1]  Rui Li,et al.  Multistage Attention ResU-Net for Semantic Segmentation of Fine-Resolution Remote Sensing Images , 2020, IEEE Geoscience and Remote Sensing Letters.

[2]  Rui Li,et al.  Multiattention Network for Semantic Segmentation of Fine-Resolution Remote Sensing Images , 2020, IEEE Transactions on Geoscience and Remote Sensing.

[3]  P. Atkinson,et al.  MACU-Net for Semantic Segmentation of Fine-Resolution Remotely Sensed Images , 2020, IEEE Geoscience and Remote Sensing Letters.

[4]  Michael Ying Yang,et al.  Real-time Semantic Segmentation with Context Aggregation Network , 2021, ISPRS Journal of Photogrammetry and Remote Sensing.

[5]  Libo Wang,et al.  Transformer Meets Convolution: A Bilateral Awareness Net-work for Semantic Segmentation of Very Fine Resolution Ur-ban Scene Images , 2021, Remote. Sens..

[6]  Marin Orsic,et al.  Efficient semantic segmentation with pyramidal fusion , 2021, Pattern Recognit..

[7]  P. Caccetta,et al.  Looking for change? Roll the Dice and demand Attention , 2020, Remote. Sens..

[8]  Kate Saenko,et al.  Real-Time Semantic Segmentation With Fast Attention , 2020, IEEE Robotics and Automation Letters.

[9]  Gang Yu,et al.  BiSeNet V2: Bilateral Network with Guided Aggregation for Real-Time Semantic Segmentation , 2020, International Journal of Computer Vision.

[10]  Rui Li,et al.  Multi-Head Linear Attention Generative Adversarial Network for Thin Cloud Removal , 2020, ArXiv.

[11]  Gui-Song Xia,et al.  Parsing very high resolution urban scene images by learning deep ConvNets with edge-aware loss , 2020, ISPRS Journal of Photogrammetry and Remote Sensing.

[12]  Nikolaos Pappas,et al.  Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention , 2020, ICML.

[13]  Xiangzheng Deng,et al.  Examining the impacts of future land use/land cover changes on climate in Punjab province, Pakistan: implications for environmental sustainability and economic growth , 2020, Environmental Science and Pollution Research.

[14]  Qinghui Liu,et al.  Dense Dilated Convolutions’ Merging Network for Land Cover Classification , 2020, IEEE Transactions on Geoscience and Remote Sensing.

[15]  Peter M. Atkinson,et al.  Scale Sequence Joint Deep Learning (SS-JDL) for land use and land cover classification , 2020, Remote Sensing of Environment.

[16]  Yang Yang,et al.  Classification of Hyperspectral Image Based on Double-Branch Dual-Attention Mechanism Network , 2019, Remote. Sens..

[17]  W. Zuo,et al.  ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Xilin Chen,et al.  Object-Contextual Representations for Semantic Segmentation , 2019, ECCV.

[19]  Peter Caccetta,et al.  ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data , 2019, ISPRS Journal of Photogrammetry and Remote Sensing.

[20]  Guisong Xia,et al.  Land-cover classification with high-resolution remote sensing images using transferable deep models , 2018, Remote Sensing of Environment.

[21]  Enhua Wu,et al.  Squeeze-and-Excitation Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Zhibin Hong,et al.  ACFNet: Attentional Class Feature Network for Semantic Segmentation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[23]  Hong Liu,et al.  Expectation-Maximization Attention Networks for Semantic Segmentation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[24]  Gen Li,et al.  DABNet: Depth-wise Asymmetric Bottleneck for Real-time Semantic Segmentation , 2019, BMVC.

[25]  Ling Shao,et al.  See More, Know More: Unsupervised Video Object Segmentation With Co-Attention Siamese Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Stephen Lin,et al.  GCNet: Non-Local Networks Meet Squeeze-Excitation Networks and Beyond , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[27]  Quoc V. Le,et al.  Attention Augmented Convolutional Networks , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[28]  Enrico Magli,et al.  Learning and Adapting Robust Features for Satellite Image Segmentation on Heterogeneous Data Sets , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[29]  Jian Yang,et al.  Selective Kernel Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Yi Sun,et al.  Problems of encoder-decoder frameworks for high-resolution remote sensing image segmentation: Structural stereotype and insufficient learning , 2019, Neurocomputing.

[31]  Roberto Cipolla,et al.  Fast-SCNN: Fast Semantic Segmentation Network , 2019, BMVC.

[32]  Xin Pan,et al.  Joint Deep Learning for land cover and land use classification , 2019, Remote Sensing of Environment.

[33]  Patrick Hostert,et al.  Intra-annual reflectance composites from Sentinel-2 and Landsat for national-scale crop and land cover mapping , 2019, Remote Sensing of Environment.

[34]  Yunchao Wei,et al.  CCNet: Criss-Cross Attention for Semantic Segmentation , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[35]  Juntang Zhuang,et al.  ShelfNet for Fast Semantic Segmentation , 2018, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[36]  Jun Fu,et al.  Dual Attention Network for Scene Segmentation , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Han Zhang,et al.  Self-Attention Generative Adversarial Networks , 2018, ICML.

[38]  Damien Arvor,et al.  Big earth observation time series analysis for monitoring Brazilian agriculture , 2018, ISPRS Journal of Photogrammetry and Remote Sensing.

[39]  Shuicheng Yan,et al.  A2-Nets: Double Attention Networks , 2018, NeurIPS.

[40]  Jingdong Wang,et al.  OCNet: Object Context Network for Scene Parsing , 2018, ArXiv.

[41]  Gang Yu,et al.  BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation , 2018, ECCV.

[42]  In-So Kweon,et al.  CBAM: Convolutional Block Attention Module , 2018, ECCV.

[43]  Jane Elith,et al.  A comparison of resampling methods for remote sensing classification and accuracy assessment , 2018 .

[44]  Ke Li,et al.  Rotation-Insensitive and Context-Augmented Object Detection in Remote Sensing Images , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[45]  George Papandreou,et al.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation , 2018, ECCV.

[46]  Lingfeng Wang,et al.  Semantic Labeling in Very High Resolution Images via a Self-Cascaded Convolutional Neural Network , 2017, ISPRS Journal of Photogrammetry and Remote Sensing.

[47]  Jiebo Luo,et al.  DOTA: A Large-Scale Dataset for Object Detection in Aerial Images , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[48]  Abhinav Gupta,et al.  Non-local Neural Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[49]  Bertrand Le Saux,et al.  Beyond RGB: Very High Resolution Urban Remote Sensing With Multimodal Deep Networks , 2017, ISPRS Journal of Photogrammetry and Remote Sensing.

[50]  Ronald Kemker,et al.  Algorithms for semantic segmentation of multispectral remote sensing imagery using deep learning , 2017, ISPRS Journal of Photogrammetry and Remote Sensing.

[51]  Garrison W. Cottrell,et al.  Understanding Convolution for Semantic Segmentation , 2017, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[52]  Uwe Stilla,et al.  Classification With an Edge: Improving Semantic Image Segmentation with Boundary Detection , 2016, ISPRS Journal of Photogrammetry and Remote Sensing.

[53]  Eduardo Romera,et al.  ERFNet: Efficient Residual Factorized ConvNet for Real-Time Semantic Segmentation , 2018, IEEE Transactions on Intelligent Transportation Systems.

[54]  Dirk Pflugmacher,et al.  Land use and land cover change in Inner Mongolia - understanding the effects of China's re-vegetation programs , 2018 .

[55]  Xiao Xiang Zhu,et al.  Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources , 2017, IEEE Geoscience and Remote Sensing Magazine.

[56]  Peijun Du,et al.  A review of supervised object-based land-cover image classification , 2017 .

[57]  George Papandreou,et al.  Rethinking Atrous Convolution for Semantic Image Segmentation , 2017, ArXiv.

[58]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[59]  Xiaogang Wang,et al.  Residual Attention Network for Image Classification , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[60]  Pierre Alliez,et al.  Convolutional Neural Networks for Large-Scale Remote-Sensing Image Classification , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[61]  Xiaogang Wang,et al.  Pyramid Scene Parsing Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[62]  Tat-Seng Chua,et al.  SCA-CNN: Spatial and Channel-Wise Attention in Convolutional Networks for Image Captioning , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[63]  François Chollet,et al.  Xception: Deep Learning with Depthwise Separable Convolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[64]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[65]  Jamie Sherrah,et al.  Fully Convolutional Networks for Dense Semantic Labelling of High-Resolution Aerial Imagery , 2016, ArXiv.

[66]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[67]  Vladlen Koltun,et al.  Multi-Scale Context Aggregation by Dilated Convolutions , 2015, ICLR.

[68]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[69]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[70]  Iasonas Kokkinos,et al.  Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs , 2014, ICLR.

[71]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[72]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[73]  Liangpei Zhang,et al.  A Hybrid Object-Oriented Conditional Random Field Classification Framework for High Spatial Resolution Remote Sensing Imagery , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[74]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[75]  J. Gardner Looking for a change. , 1997, Modern Healthcare.

[76]  Peng Gong,et al.  A comparison of spatial feature extraction algorithms for land-use classification with SPOT HRV data , 1992 .

[77]  C. Tucker Red and photographic infrared linear combinations for monitoring vegetation , 1979 .