Pansharpening via Detail Injection Based Convolutional Neural Networks

Pansharpening aims to fuse a multispectral (MS) image with an associated panchromatic (PAN) image, producing a composite image with the spectral resolution of the former and the spatial resolution of the latter. Traditional pansharpening methods can be ascribed to a unified detail injection context, which views the injected MS details as the integration of PAN details and bandwise injection gains. In this paper, we design a new detail injection based convolutional neural network (DiCNN) framework for pansharpening with the MS details being directly formulated in end-to-end manners, where the first detail injection based CNN (DiCNN1) mines MS details through the PAN image and the MS image, and the second one (DiCNN2) utilizes only the PAN image. The main advantage of the proposed DiCNNs is that they provide explicit physical interpretations and can achieve fast convergence while achieving high pansharpening quality. Furthermore, the effectiveness of the proposed approaches is also analyzed from a relatively theoretical point of view. Our methods are evaluated via experiments on real MS image datasets, achieving excellent performance when compared to other state-of-the-art methods.

[1]  Te-Ming Tu,et al.  A new look at IHS-like image fusion methods , 2001, Inf. Fusion.

[2]  Xavier Otazu,et al.  Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods , 2005, IEEE Transactions on Geoscience and Remote Sensing.

[3]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[4]  Xiaoou Tang,et al.  Learning a Deep Convolutional Network for Image Super-Resolution , 2014, ECCV.

[5]  Qi Wei.  Bayesian Fusion of Multi-band Images: A Powerful Tool for Super-resolution. (Fusion Bayesieene des multi-bandes Images: Un outil puissant pour la Super-résolution) , 2015 .

[6]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[7]  Xiuping Jia,et al.  Deep Feature Extraction and Classification of Hyperspectral Images Based on Convolutional Neural Networks , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[8]  Yoshua Bengio,et al.  How transferable are features in deep neural networks? , 2014, NIPS.

[9]  L. Wald,et al.  Fusion of high spatial and spectral resolution images : The ARSIS concept and its implementation , 2000 .

[10]  Naoto Yokoya,et al.  Hyperspectral Pansharpening: A Review , 2015, IEEE Geoscience and Remote Sensing Magazine.

[11]  Luciano Alparone,et al.  A global quality measurement of pan-sharpened multispectral imagery , 2004, IEEE Geoscience and Remote Sensing Letters.

[12]  Simon Haykin,et al.  Neural Networks and Learning Machines , 2010 .

[13]  Andrea Garzelli,et al.  Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis , 2002, IEEE Trans. Geosci. Remote. Sens..

[14]  Andrea Garzelli,et al.  Optimal MMSE Pan Sharpening of Very High Resolution Multispectral Images , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[15]  Jocelyn Chanussot,et al.  Comparison of Pansharpening Algorithms: Outcome of the 2006 GRS-S Data-Fusion Contest , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[16]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Luciano Alparone,et al.  MTF-tailored Multiscale Fusion of High-resolution MS and Pan Imagery , 2006 .

[18]  Andrea Vedaldi,et al.  MatConvNet: Convolutional Neural Networks for MATLAB , 2014, ACM Multimedia.

[19]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[20]  Jocelyn Chanussot,et al.  Contrast and Error-Based Fusion Schemes for Multispectral Image Pansharpening , 2014, IEEE Geoscience and Remote Sensing Letters.

[21]  Fan Zhang,et al.  Deep Convolutional Neural Networks for Hyperspectral Image Classification , 2015, J. Sensors.

[22]  Jocelyn Chanussot,et al.  A Critical Comparison Among Pansharpening Algorithms , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[23]  Roger L. King,et al.  An Efficient Pan-Sharpening Method via a Combined Adaptive PCA Approach and Contourlets , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[24]  Xiangmin Xu,et al.  Single Image Super-Resolution Using Multi-Scale Convolutional Neural Network , 2017, PCM.

[25]  Yizhou Rao,et al.  A residual convolutional neural network for pan-shaprening , 2017, 2017 International Workshop on Remote Sensing with Intelligent Processing (RSIP).

[26]  Davide Cozzolino,et al.  Pansharpening by Convolutional Neural Networks , 2016, Remote. Sens..

[27]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[28]  Kunihiko Fukushima,et al.  Neocognitron: A hierarchical neural network capable of visual pattern recognition , 1988, Neural Networks.

[29]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[30]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[31]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Bruno Aiazzi,et al.  Improving Component Substitution Pansharpening Through Multivariate Regression of MS $+$Pan Data , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[33]  L. Alparone,et al.  An MTF-based spectral distortion minimizing model for pan-sharpening of very high resolution multispectral images of urban areas , 2003, 2003 2nd GRSS/ISPRS Joint Workshop on Remote Sensing and Data Fusion over Urban Areas.

[34]  Jean-Yves Tourneret,et al.  Bayesian Fusion of Multi-Band Images , 2013, IEEE Journal of Selected Topics in Signal Processing.

[35]  Jocelyn Chanussot,et al.  Synthesis of Multispectral Images to High Spatial Resolution: A Critical Review of Fusion Methods Based on Remote Sensing Physics , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[36]  Alan R. Gillespie,et al.  Color enhancement of highly correlated images. II. Channel ratio and “chromaticity” transformation techniques , 1987 .

[37]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Kiyun Yu,et al.  A New Adaptive Component-Substitution-Based Satellite Image Fusion by Using Partial Replacement , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[39]  Jocelyn Chanussot,et al.  Indusion: Fusion of Multispectral and Panchromatic Images Using the Induction Scaling Technique , 2008, IEEE Geoscience and Remote Sensing Letters.

[40]  Liangpei Zhang,et al.  Boosting the Accuracy of Multispectral Image Pansharpening by Learning a Deep Residual Network , 2017, IEEE Geoscience and Remote Sensing Letters.

[41]  J. Boardman,et al.  Discrimination among semi-arid landscape endmembers using the Spectral Angle Mapper (SAM) algorithm , 1992 .

[42]  L. Wald,et al.  Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images , 1997 .

[43]  P. Chavez,et al.  Extracting spectral contrast in landsat thematic mapper image data using selective principal component analysis , 1989 .

[44]  Yann LeCun,et al.  The Loss Surfaces of Multilayer Networks , 2014, AISTATS.

[45]  V. K. Shettigara,et al.  A generalized component substitution technique for spatial enhancement of multispectral images using , 1992 .

[46]  Léon Bottou,et al.  Large-Scale Machine Learning with Stochastic Gradient Descent , 2010, COMPSTAT.

[47]  Liangpei Zhang,et al.  A Multiscale and Multidepth Convolutional Neural Network for Remote Sensing Imagery Pan-Sharpening , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[48]  Chulhee Lee,et al.  Fast and Efficient Panchromatic Sharpening , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[49]  B. Silverman,et al.  The Stationary Wavelet Transform and some Statistical Applications , 1995 .

[50]  J. Zhou,et al.  A wavelet transform method to merge Landsat TM and SPOT panchromatic data , 1998 .

[51]  Lucien Wald,et al.  Data Fusion. Definitions and Architectures - Fusion of Images of Different Spatial Resolutions , 2002 .

[52]  Aleksandra Pizurica,et al.  Processing of Multiresolution Thermal Hyperspectral and Digital Color Data: Outcome of the 2014 IEEE GRSS Data Fusion Contest , 2015, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.