Boosting the Accuracy of Multispectral Image Pansharpening by Learning a Deep Residual Network

In the field of multispectral (MS) and panchromatic image fusion (pansharpening), the impressive effectiveness of deep neural networks has recently been employed to overcome the drawbacks of the traditional linear models and boost the fusion accuracy. However, the existing methods are mainly based on simple and flat networks with relatively shallow architectures, which severely limits their performance. In this letter, the concept of residual learning is introduced to form a very deep convolutional neural network to make the full use of the high nonlinearity of the deep learning models. Through both quantitative and visual assessments on a large number of high-quality MS images from various sources, it is confirmed that the proposed model is superior to all the mainstream algorithms included in the comparison, and achieves the highest spatial–spectral unified accuracy.

[1]  Lei Zhang,et al.  Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising , 2016, IEEE Transactions on Image Processing.

[2]  Jie Li,et al.  Noise Removal From Hyperspectral Image With Joint Spectral–Spatial Distributed Sparse Representation , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[3]  Boris Polyak.  Some methods of speeding up the convergence of iteration methods , 1964 .

[4]  Wenzhong Shi,et al.  Fusion of Sentinel-2 images , 2016 .

[5]  Qi Wang,et al.  Salient Band Selection for Hyperspectral Image Classification via Manifold Ranking , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[6]  Hong Sun,et al.  Unsupervised Feature Learning Via Spectral Clustering of Multidimensional Patches for Remotely Sensed Scene Classification , 2015, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[7]  Xuelong Li,et al.  Saliency Detection by Multiple-Instance Learning , 2013, IEEE Transactions on Cybernetics.

[8]  Wenzhong Shi,et al.  Area-to-point regression kriging for pan-sharpening , 2016 .

[9]  Davide Cozzolino,et al.  Pansharpening by Convolutional Neural Networks , 2016, Remote. Sens..

[10]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[11]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Pingkun Yan,et al.  Visual Saliency by Selective Contrast , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[13]  Hongyi Liu,et al.  A New Pan-Sharpening Method With Deep Neural Networks , 2015, IEEE Geoscience and Remote Sensing Letters.

[14]  Hod Lipson,et al.  Understanding Neural Networks Through Deep Visualization , 2015, ArXiv.

[15]  Liangpei Zhang,et al.  Two-Step Sparse Coding for the Pan-Sharpening of Remote Sensing Images , 2014, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[16]  Liangpei Zhang,et al.  An Integrated Framework for the Spatio–Temporal–Spectral Fusion of Remote Sensing Images , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[17]  J. G. Liu,et al.  Smoothing Filter-based Intensity Modulation : a spectral preserve image fusion technique for improving spatial details , 2001 .

[18]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Pascal Vincent,et al.  Visualizing Higher-Layer Features of a Deep Network , 2009 .

[20]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Kiyun Yu,et al.  A New Adaptive Component-Substitution-Based Satellite Image Fusion by Using Partial Replacement , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[22]  Liang-pei Zhang,et al.  Integrated fusion of multi-scale polar-orbiting and geostationary satellite observations for the mapping of high spatial and temporal resolution land surface temperature , 2015 .

[23]  Liangpei Zhang,et al.  Adaptive Multiple-Frame Image Super-Resolution Based on U-Curve , 2010, IEEE Transactions on Image Processing.

[24]  Xavier Otazu,et al.  Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods , 2005, IEEE Transactions on Geoscience and Remote Sensing.

[25]  Johannes R. Sveinsson,et al.  Multispectral and Hyperspectral Image Fusion Using a 3-D-Convolutional Neural Network , 2017, IEEE Geoscience and Remote Sensing Letters.

[26]  Luciano Alparone,et al.  MTF-tailored Multiscale Fusion of High-resolution MS and Pan Imagery , 2006 .