A Cross-Direction and Progressive Network for Pan-Sharpening

In this paper, we propose a cross-direction and progressive network, termed CPNet, to solve the pan-sharpening problem. The full processing of information is the main characteristic of our model, which is reflected as follows: on the one hand, we process the source images in a cross-direction manner to obtain the source images of different scales as the input of the fusion modules at different stages, which maximizes the usage of multi-scale information in the source images; on the other hand, the progressive reconstruction loss is designed to boost the training of our network and avoid partial inactivation, while maintaining the consistency of the fused result with the ground truth. Since the extraction of the information from the source images and the reconstruction of the fused image is based on the entire image rather than a single type of information, there is little loss of partial spatial or spectral information due to insufficient information processing. Extensive experiments, including qualitative and quantitative comparisons demonstrate that our model can maintain more spatial and spectral information compared to the state-of-the-art pan-sharpening methods.

[1]  Xiaomin Yang,et al.  Two-Path Network with Feedback Connections for Pan-Sharpening in Remote Sensing , 2020, Remote. Sens..

[2]  Liangpei Zhang,et al.  Boosting the Accuracy of Multispectral Image Pansharpening by Learning a Deep Residual Network , 2017, IEEE Geoscience and Remote Sensing Letters.

[3]  Bin Xie,et al.  Revealing Implicit Assumptions of the Component Substitution Pansharpening Methods , 2017, Remote. Sens..

[4]  Te-Ming Tu,et al.  A new look at IHS-like image fusion methods , 2001, Inf. Fusion.

[5]  Qingjie Liu,et al.  PSGAN: A Generative Adversarial Network for Remote Sensing Image Pan-Sharpening , 2020, IEEE Transactions on Geoscience and Remote Sensing.

[6]  Hassan Ghassemian,et al.  A review of remote sensing image fusion methods , 2016, Inf. Fusion.

[7]  J. Strobl,et al.  Object-Oriented Image Processing in an Integrated GIS/Remote Sensing Environment and Perspectives for Environmental Applications , 2000 .

[8]  Gintautas Palubinskas,et al.  Joint Quality Measure for Evaluation of Pansharpening Accuracy , 2015, Remote. Sens..

[9]  S. Sides,et al.  Comparison of three different methods to merge multiresolution and multispectral data: Landsat TM and SPOT panchromatic , 1991 .

[10]  Qin Liu,et al.  Hybrid Attention Based Residual Network for Pansharpening , 2021, Remote. Sens..

[11]  Jocelyn Chanussot,et al.  Comparison of Pansharpening Algorithms: Outcome of the 2006 GRS-S Data-Fusion Contest , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[12]  J. Zhou,et al.  A wavelet transform method to merge Landsat TM and SPOT panchromatic data , 1998 .

[13]  Jinying Zhong,et al.  Remote Sensing Image Fusion with Convolutional Neural Network , 2016 .

[14]  Davide Cozzolino,et al.  Target-Adaptive CNN-Based Pansharpening , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[15]  Bo Du,et al.  Hyperspectral Remote Sensing Image Subpixel Target Detection Based on Supervised Metric Learning , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[16]  Jocelyn Chanussot,et al.  A Two-Stream Multiscale Deep Learning Architecture for Pan-Sharpening , 2020, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[17]  Jiemin Wu,et al.  Deep Self-Learning Network for Adaptive Pansharpening , 2019, Remote. Sens..

[18]  Chen Chen,et al.  Pan-GAN: An unsupervised pan-sharpening method for remote sensing image fusion , 2020, Inf. Fusion.

[19]  Ying Li,et al.  Going Deeper with Densely Connected Convolutional Neural Networks for Multispectral Pansharpening , 2019, Remote. Sens..

[20]  Jocelyn Chanussot,et al.  Synthesis of Multispectral Images to High Spatial Resolution: A Critical Review of Fusion Methods Based on Remote Sensing Physics , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[21]  Aaron C. Courville,et al.  Generative adversarial networks , 2020 .

[22]  Jaewan Choi,et al.  A Hybrid Pansharpening Algorithm of VHR Satellite Images that Employs Injection Gains Based on NDVI to Reduce Computational Costs , 2017, Remote. Sens..

[23]  Jiayi Ma,et al.  Variational Pansharpening by Exploiting Cartoon-Texture Similarities , 2021, IEEE Transactions on Geoscience and Remote Sensing.

[24]  Jiayi Ma,et al.  Image fusion meets deep learning: A survey and perspective , 2021, Inf. Fusion.

[25]  Naoto Yokoya,et al.  Coupled Nonnegative Matrix Factorization Unmixing for Hyperspectral and Multispectral Data Fusion , 2012, IEEE Transactions on Geoscience and Remote Sensing.

[26]  Hui Li,et al.  DenseFuse: A Fusion Approach to Infrared and Visible Images , 2018, IEEE Transactions on Image Processing.

[27]  M. Ghadjati,et al.  A novel iterative PCA–based pansharpening method , 2018, Remote Sensing Letters.

[28]  Christian Germain,et al.  Generative Adversarial Network for Pansharpening With Spectral and Spatial Discriminators , 2021, IEEE Transactions on Geoscience and Remote Sensing.

[29]  Davide Cozzolino,et al.  Pansharpening by Convolutional Neural Networks , 2016, Remote. Sens..

[30]  L. Wald,et al.  Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images , 1997 .

[31]  Keiron O'Shea,et al.  An Introduction to Convolutional Neural Networks , 2015, ArXiv.

[32]  Kiyun Yu,et al.  A New Adaptive Component-Substitution-Based Satellite Image Fusion by Using Partial Replacement , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[33]  Giuseppe Scarpa,et al.  A Detail-Preserving Cross-Scale Learning Strategy for CNN-Based Pansharpening , 2020, Remote. Sens..

[34]  Leyuan Fang,et al.  Residual Encoder–Decoder Conditional Generative Adversarial Network for Pansharpening , 2020, IEEE Geoscience and Remote Sensing Letters.

[35]  Roger L. King,et al.  An Efficient Pan-Sharpening Method via a Combined Adaptive PCA Approach and Contourlets , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[36]  Renwei Dian,et al.  Regularizing Hyperspectral and Multispectral Image Fusion by CNN Denoiser , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[37]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[38]  Hao Zhang,et al.  SDPNet: A Deep Network for Pan-Sharpening With Enhanced Information Representation , 2021, IEEE Transactions on Geoscience and Remote Sensing.

[39]  Gong Zhang,et al.  Improving Remote Sensing Image Super-Resolution Mapping Based on the Spatial Attraction Model by Utilizing the Pansharpening Technique , 2019, Remote. Sens..

[40]  Jiayi Ma,et al.  GTP-PNet: A residual learning network based on gradient transformation prior for pansharpening , 2021 .

[41]  Consuelo Gonzalo-Martín,et al.  Scale-Aware Pansharpening Algorithm for Agricultural Fragmented Landscapes , 2016, Remote. Sens..

[42]  Changsheng Zhou,et al.  PercepPan: Towards Unsupervised Pan-Sharpening Based on Perceptual Loss , 2020, Remote. Sens..

[43]  Laura Igual,et al.  A Variational Model for P+XS Image Fusion , 2006, International Journal of Computer Vision.