Pansharpening via Unsupervised Convolutional Neural Networks

Pansharpening is normally utilized to take full advantage of all the available spectral and spatial information that are derived from a low-spatial-resolution multispectral (MS) image and its associated high-spatial-resolution (HR) panchromatic (PAN) image, respectively, producing a fused MS image with high spectral and spatial resolutions. Many methods have been recently developed based on convolutional neural networks (CNNs) for the pansharpening task, but most of them still have some drawbacks: 1) The information cannot efficiently flow in their simple stacked convolutional architectures, thereby hindering the representation ability of the networks. 2) They are commonly trained using supervised learning, which does not only require an extra effort to produce the simulated training data, but can also lead to scale-related problems in the fusion results. In this article, we propose a novel unsupervised CNN-based pansharpening method to overcome these limitations. Specifically, we design an iterative network architecture, in which a PAN-guided strategy and a set of skip connections are adopted to continuously extract and fuse the features from the input, thus enhancing the information reuse and transmission. Besides, we propose a new loss function for unsupervised training in which the relationships between the input MS and PAN images and the fused MS image are used to design the spatial constrains and spectral consistency, respectively. The typical quality index with no-reference is also added to this function to further adjust the spectral and spatial qualities. The designed loss function allows the network to be learned only on input images, without any hand-crafted labels (reference HR MS image). We evaluated the effectiveness of our designed network architecture and the combined loss function, and the experiments testify that our unsupervised strategy can also obtain promising results with minor spectral and spatial distortions compared with other traditional and supervised methods.

[1]  Lucien Wald,et al.  Quality of high resolution synthesised images: Is there a simple criterion ? , 2000 .

[2]  Myeong-Ryong Nam,et al.  Fusion of multispectral and panchromatic Satellite images using the curvelet transform , 2005, IEEE Geoscience and Remote Sensing Letters.

[3]  V. K. Shettigara,et al.  A generalized component substitution technique for spatial enhancement of multispectral images using , 1992 .

[4]  Te-Ming Tu,et al.  A fast intensity-hue-saturation fusion technique with spectral adjustment for IKONOS imagery , 2004, IEEE Geoscience and Remote Sensing Letters.

[5]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[6]  Shu Zhang,et al.  Scene-Adaptive Remote Sensing Image Super-Resolution Using a Multiscale Attention Network , 2020, IEEE Transactions on Geoscience and Remote Sensing.

[7]  Qian Du,et al.  DDLPS: Detail-Based Deep Laplacian Pansharpening for Hyperspectral Imagery , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[8]  Jocelyn Chanussot,et al.  Hyperspectral Pansharpening Using Deep Prior and Dual Attention Residual Network , 2020, IEEE Transactions on Geoscience and Remote Sensing.

[9]  Yifan Zhang,et al.  A Bayesian Restoration Approach for Hyperspectral Images , 2012, IEEE Transactions on Geoscience and Remote Sensing.

[10]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[11]  Luciano Alparone,et al.  A global quality measurement of pan-sharpened multispectral imagery , 2004, IEEE Geoscience and Remote Sensing Letters.

[12]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Debing Zhang,et al.  Hyperspectral Image Super-Resolution With Optimized RGB Guidance , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Kiyun Yu,et al.  A New Adaptive Component-Substitution-Based Satellite Image Fusion by Using Partial Replacement , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[15]  Andrea Garzelli,et al.  A Review of Image Fusion Algorithms Based on the Super-Resolution Paradigm , 2016, Remote. Sens..

[16]  Jue Zhang,et al.  A New Saliency-Driven Fusion Method Based on Complex Wavelet Transform for Remote Sensing Images , 2017, IEEE Geoscience and Remote Sensing Letters.

[17]  Jun Li,et al.  Pansharpening via Detail Injection Based Convolutional Neural Networks , 2018, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[18]  W. J. Carper,et al.  The use of intensity-hue-saturation transformations for merging SPOT panchromatic and multispectral image data , 1990 .

[19]  J. Zhou,et al.  A wavelet transform method to merge Landsat TM and SPOT panchromatic data , 1998 .

[20]  L. Wald,et al.  Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images , 1997 .

[21]  Jocelyn Chanussot,et al.  Contrast and Error-Based Fusion Schemes for Multispectral Image Pansharpening , 2014, IEEE Geoscience and Remote Sensing Letters.

[22]  Jocelyn Chanussot,et al.  A Critical Comparison Among Pansharpening Algorithms , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[23]  Zhou Wang,et al.  Image Quality Assessment: From Error Measurement to Structural Similarity , 2004 .

[24]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[25]  Roger L. King,et al.  An Efficient Pan-Sharpening Method via a Combined Adaptive PCA Approach and Contourlets , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[26]  Jon Y. Hardeberg,et al.  Deep Hyperspectral Prior: Denoising, Inpainting, Super-Resolution , 2019, ArXiv.

[27]  Andrea Garzelli,et al.  Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis , 2002, IEEE Trans. Geosci. Remote. Sens..

[28]  Davide Cozzolino,et al.  Pansharpening by Convolutional Neural Networks , 2016, Remote. Sens..

[29]  Andrea Garzelli,et al.  Optimal MMSE Pan Sharpening of Very High Resolution Multispectral Images , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[30]  Jinying Zhong,et al.  Remote Sensing Image Fusion with Convolutional Neural Network , 2016 .

[31]  Xavier Otazu,et al.  Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods , 2005, IEEE Transactions on Geoscience and Remote Sensing.

[32]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Jocelyn Chanussot,et al.  HyperPNN: Hyperspectral Pansharpening via Spectrally Predictive Convolutional Neural Networks , 2019, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[34]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[35]  Luciano Alparone,et al.  MTF-tailored Multiscale Fusion of High-resolution MS and Pan Imagery , 2006 .

[36]  Hassan Ghassemian,et al.  Combining the spectral PCA and spatial PCA fusion methods by an optimal filter , 2016, Inf. Fusion.

[37]  Jocelyn Chanussot,et al.  Comparison of Pansharpening Algorithms: Outcome of the 2006 GRS-S Data-Fusion Contest , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[38]  Hassan Ghassemian,et al.  Multi spectral image fusion by deep convolutional neural network and new spectral loss function , 2018 .

[39]  Lucien Wald,et al.  Analysis of Changes in Quality Assessment with Scale , 2006, 2006 9th International Conference on Information Fusion.

[40]  Shuyuan Yang,et al.  Convolution Structure Sparse Coding for Fusion of Panchromatic and Multispectral Images , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[41]  Tao Li,et al.  A Variational Pan-Sharpening Method Based on Spatial Fractional-Order Geometry and Spectral–Spatial Low-Rank Priors , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[42]  Jocelyn Chanussot,et al.  Indusion: Fusion of Multispectral and Panchromatic Images Using the Induction Scaling Technique , 2008, IEEE Geoscience and Remote Sensing Letters.

[43]  Yun Fu,et al.  Residual Dense Network for Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[44]  S. Baronti,et al.  Multispectral and panchromatic data fusion assessment without reference , 2008 .

[45]  A. Bovik,et al.  A universal image quality index , 2002, IEEE Signal Processing Letters.

[46]  Baohua Qiang,et al.  A novel adaptive fast IHS transform fusion method driven by regional spectral characteristics for Gaofen-2 imagery , 2020 .

[47]  L. Wald,et al.  Fusion of high spatial and spectral resolution images : The ARSIS concept and its implementation , 2000 .

[48]  Richard Bamler,et al.  Exploiting Joint Sparsity for Pansharpening: The J-SparseFI Algorithm , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[49]  Zhenfeng Shao,et al.  Remote Sensing Image Fusion With Deep Convolutional Neural Network , 2018, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[50]  Shu-Tao Xia,et al.  Second-Order Attention Network for Single Image Super-Resolution , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Bruno Aiazzi,et al.  Improving Component Substitution Pansharpening Through Multivariate Regression of MS $+$Pan Data , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[52]  Liangpei Zhang,et al.  Boosting the Accuracy of Multispectral Image Pansharpening by Learning a Deep Residual Network , 2017, IEEE Geoscience and Remote Sensing Letters.

[53]  J. Boardman,et al.  Discrimination among semi-arid landscape endmembers using the Spectral Angle Mapper (SAM) algorithm , 1992 .