ICIF-Net: Intra-Scale Cross-Interaction and Inter-Scale Feature Fusion Network for Bitemporal Remote Sensing Images Change Detection

Change detection (CD) of remote sensing (RS) images has enjoyed remarkable success by virtue of convolutional neural networks (CNNs) with promising discriminative capabilities. However, CNNs lack the capability of modeling long-range dependencies in bitemporal image pairs, resulting in inferior identifiability against the same semantic targets yet with varying features. The recently thriving Transformer, on the contrary, is warranted, for practice, with global receptive fields. To jointly harvest the local-global features and circumvent the misalignment issues caused by step-by-step downsampling operations in traditional backbone networks, we propose an intra-scale cross-interaction and inter-scale feature fusion network (ICIF-Net), explicitly tapping the potential of integrating CNN and Transformer. In particular, the local features and global features, respectively, extracted by CNN and Transformer, are interactively communicated at the same spatial resolution using a linearized Conv Attention module, which motivates the counterpart to glimpse the representation of another branch while preserving its own features. In addition, with the introduction of two attention-based inter-scale fusion schemes, including mask-based aggregation and spatial alignment (SA), information integration is enforced at different resolutions. Finally, the integrated features are fed into a conventional change prediction head to generate the output. Extensive experiments conducted on four CD datasets of bitemporal (RS) images demonstrate that our ICIF-Net surpasses the other state-of-the-art (SOTA) approaches.

[1]  Licheng Jiao,et al.  An Unsupervised Remote Sensing Change Detection Method Based on Multiscale Graph Convolutional Network and Metric Learning , 2021, IEEE Transactions on Geoscience and Remote Sensing.

[2]  Lianru Gao,et al.  SpectralFormer: Rethinking Hyperspectral Image Classification With Transformers , 2021, IEEE Transactions on Geoscience and Remote Sensing.

[3]  Liangpei Zhang,et al.  A Deeply Supervised Attention Metric-Based Network and an Open Aerial Image Dataset for Remote Sensing Change Detection , 2021, IEEE Transactions on Geoscience and Remote Sensing.

[4]  P. Luo,et al.  PVT v2: Improved baselines with Pyramid Vision Transformer , 2021, Computational Visual Media.

[5]  Zhenwei Shi,et al.  Remote Sensing Image Change Detection With Transformers , 2021, IEEE Transactions on Geoscience and Remote Sensing.

[6]  Jinyuan Shao,et al.  SNUNet-CD: A Densely Connected Siamese Network for Change Detection of VHR Images , 2021, IEEE Geoscience and Remote Sensing Letters.

[7]  C. Ding,et al.  End-to-End Method With Transformer for 3-D Detection of Oil Tank From Single SAR Image , 2022, IEEE Transactions on Geoscience and Remote Sensing.

[8]  Cong Bai,et al.  3D Octave and 2D Vanilla Mixed Convolutional Neural Network for Hyperspectral Image Classification with Limited Samples , 2021, Remote. Sens..

[9]  A. Dosovitskiy,et al.  Do Vision Transformers See Like Convolutional Neural Networks? , 2021, NeurIPS.

[10]  Yakoub Bazi,et al.  Deep Vision Transformers for Remote Sensing Scene Classification , 2021, 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS.

[11]  Lin Lei,et al.  Iterative Robust Graph for Unsupervised Change Detection of Heterogeneous Remote Sensing Images , 2021, IEEE Transactions on Image Processing.

[12]  Liang Gao,et al.  STransFuse: Fusing Swin Transformer and Convolutional Neural Network for Remote Sensing Image Semantic Segmentation , 2021, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[13]  Trevor Darrell,et al.  Early Convolutions Help Transformers See Better , 2021, NeurIPS.

[14]  Wenzhong Shi,et al.  Landslide Recognition by Deep Convolutional Neural Network and Change Detection , 2021, IEEE Transactions on Geoscience and Remote Sensing.

[15]  Zhuowen Tu,et al.  Co-Scale Conv-Attentional Image Transformers , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[16]  N. Codella,et al.  CvT: Introducing Convolutions to Vision Transformers , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[17]  Huchuan Lu,et al.  Transformer Tracking , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Tao Xiang,et al.  Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Yanjun Qi,et al.  General Multi-label Image Classification with Transformers , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Lorenzo Bruzzone,et al.  SemiCDNet: A Semisupervised Convolutional Neural Network for Change Detection in High Resolution Remote-Sensing Images , 2020, IEEE Transactions on Geoscience and Remote Sensing.

[21]  Cong Bai,et al.  Hyperspectral Image Classification Using Mixed Convolutions and Covariance Pooling , 2020, IEEE Trans. Geosci. Remote. Sens..

[22]  Xiaomeng Zhang,et al.  Building Change Detection for Remote Sensing Images Using a Dual-Task Constrained Deep Siamese Convolutional Network Model , 2019, IEEE Geoscience and Remote Sensing Letters.

[23]  Weishi Zheng,et al.  Spectral-Spatial Transformer Network for Hyperspectral Image Classification: A Factorized Architecture Search Framework , 2021, IEEE Transactions on Geoscience and Remote Sensing.

[24]  Stephen Lin,et al.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[25]  Peng Yue,et al.  A deeply supervised image fusion network for change detection in high resolution bi-temporal remote sensing images , 2020 .

[26]  Chen Sun,et al.  Multi-modal Transformer for Video Retrieval , 2020, ECCV.

[27]  Mark Chen,et al.  Generative Pretraining From Pixels , 2020, ICML.

[28]  Nicolas Usunier,et al.  End-to-End Object Detection with Transformers , 2020, ECCV.

[29]  Hao Chen,et al.  A Spatial-Temporal Attention-Based Method and a New Dataset for Remote Sensing Image Change Detection , 2020, Remote. Sens..

[30]  J. Rogan,et al.  A near-real-time approach for monitoring forest disturbance using Landsat time series: stochastic continuous change detection , 2020, Remote Sensing of Environment.

[31]  Wenzhong Shi,et al.  A Feature Difference Convolutional Neural Network-Based Change Detection Method , 2020, IEEE Transactions on Geoscience and Remote Sensing.

[32]  Xuelong Li,et al.  Stacked Fisher autoencoder for SAR change detection , 2019, Pattern Recognit..

[33]  Gholamreza Akbarizadeh,et al.  Change detection in SAR images using deep belief network: a new training approach based on morphological images , 2019, IET Image Process..

[34]  Menglong Yan,et al.  Triplet-Based Semantic Relation Learning for Aerial Remote Sensing Image Change Detection , 2019, IEEE Geoscience and Remote Sensing Letters.

[35]  Alexandre Boulch,et al.  Multitask learning for large-scale semantic change detection , 2018, Comput. Vis. Image Underst..

[36]  Xiao Xiang Zhu,et al.  Learning Spectral-Spatial-Temporal Features via a Recurrent Convolutional Neural Network for Change Detection in Multispectral Imagery , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[37]  Meng Lu,et al.  Fully Convolutional Networks for Multisource Building Extraction From an Open Aerial and Satellite Imagery Data Set , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[38]  Yang Dam Eo,et al.  Fusion of SAR and Multispectral Images Using Random Forest Regression for Change Detection , 2018, ISPRS Int. J. Geo Inf..

[39]  Alexandre Boulch,et al.  Fully Convolutional Siamese Networks for Change Detection , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[40]  Alexandre Boulch,et al.  Urban Change Detection for Multispectral Earth Observation Using Convolutional Neural Networks , 2018, IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium.

[41]  Raffay Hamid,et al.  Toward a Generalizable Image Representation for Large-Scale Change Detection: Application to Generic Damage Analysis , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[42]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  G Verdier,et al.  Adaptive Mahalanobis Distance and $k$ -Nearest Neighbor Rule for Fault Detection in Semiconductor Manufacturing , 2011, IEEE Transactions on Semiconductor Manufacturing.

[44]  Turgay Çelik,et al.  Unsupervised Change Detection in Satellite Images Using Principal Component Analysis and $k$-Means Clustering , 2009, IEEE Geoscience and Remote Sensing Letters.

[45]  Ola Ahlqvist,et al.  Extending post-classification change detection using semantic similarity metrics to overcome class heterogeneity: A study of 1992 and 2001 U.S. National Land Cover Database changes , 2008 .

[46]  Arthur Gretton,et al.  An online support vector machine for abnormal events detection , 2006, Signal Process..

[47]  R. Jackson Spectral indices in N-Space , 1983 .

[48]  W. Todd Urban and regional land use change detected by using Landsat data , 1977 .