Pan-Sharpening Using an Efficient Bidirectional Pyramid Network

Pan-sharpening is an important preprocessing step for remote sensing image processing tasks; it fuses a low-resolution multispectral image and a high-resolution (HR) panchromatic (PAN) image to reconstruct a HR multispectral (MS) image. This paper introduces a new end-to-end bidirectional pyramid network for pan-sharpening. The overall structure of the proposed network is a bidirectional pyramid, which permits the network to process MS and PAN images in two separate branches level by level. At each level of the network, spatial details extracted from the PAN image are injected into the upsampled MS image to reconstruct the pan-sharpened image from coarse resolution to fine resolution. Subpixel convolutional layers and the enhanced residual blocks are used to make the network efficient. Comparison of the results obtained with our proposed method and the results using other widely used state-of-the-art approaches confirms that our proposed method outperforms the others in visual appearance and objective indexes.

[1]  Davide Cozzolino,et al.  Target-Adaptive CNN-Based Pansharpening , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[2]  Bruno Aiazzi,et al.  Improving Component Substitution Pansharpening Through Multivariate Regression of MS $+$Pan Data , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[3]  Xavier Otazu,et al.  Multiresolution-based image fusion with additive wavelet decomposition , 1999, IEEE Trans. Geosci. Remote. Sens..

[4]  Xiaoou Tang,et al.  Learning a Deep Convolutional Network for Image Super-Resolution , 2014, ECCV.

[5]  L. Wald,et al.  Fusion of high spatial and spectral resolution images : The ARSIS concept and its implementation , 2000 .

[6]  Xiaoou Tang,et al.  Accelerating the Super-Resolution Convolutional Neural Network , 2016, ECCV.

[7]  Kyoung Mu Lee,et al.  Deeply-Recursive Convolutional Network for Image Super-Resolution , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Hongyi Liu,et al.  A New Pan-Sharpening Method With Deep Neural Networks , 2015, IEEE Geoscience and Remote Sensing Letters.

[9]  Shutao Li,et al.  A New Pan-Sharpening Method Using a Compressed Sensing Technique , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[10]  Luciano Alparone,et al.  Remote sensing image fusion using the curvelet transform , 2007, Inf. Fusion.

[11]  Hamid Reza Shahdoosti,et al.  Multimodal image fusion using sparse representation classification in tetrolet domain , 2018, Digit. Signal Process..

[12]  Davide Cozzolino,et al.  Pansharpening by Convolutional Neural Networks , 2016, Remote. Sens..

[13]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Liangpei Zhang,et al.  A Multiscale and Multidepth Convolutional Neural Network for Remote Sensing Imagery Pan-Sharpening , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[15]  Jon Atli Benediktsson,et al.  Pansharpening With Matting Model , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[16]  Liangpei Zhang,et al.  Boosting the Accuracy of Multispectral Image Pansharpening by Learning a Deep Residual Network , 2017, IEEE Geoscience and Remote Sensing Letters.

[17]  J. Boardman,et al.  Discrimination among semi-arid landscape endmembers using the Spectral Angle Mapper (SAM) algorithm , 1992 .

[18]  Enhong Chen,et al.  Image Denoising and Inpainting with Deep Neural Networks , 2012, NIPS.

[19]  Luciano Alparone,et al.  A global quality measurement of pan-sharpened multispectral imagery , 2004, IEEE Geoscience and Remote Sensing Letters.

[20]  L. Wald,et al.  Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images , 1997 .

[21]  Te-Ming Tu,et al.  A fast intensity-hue-saturation fusion technique with spectral adjustment for IKONOS imagery , 2004, IEEE Geoscience and Remote Sensing Letters.

[22]  P. Chavez,et al.  Extracting spectral contrast in landsat thematic mapper image data using selective principal component analysis , 1989 .

[23]  Hamid Reza Shahdoosti,et al.  Pansharpening of Clustered MS and Pan Images Considering Mixed Pixels , 2017, IEEE Geoscience and Remote Sensing Letters.

[24]  Shunlin Liang,et al.  Pan-sharpening using a guided filter , 2016 .

[25]  Roger L. King,et al.  An Efficient Pan-Sharpening Method via a Combined Adaptive PCA Approach and Contourlets , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[26]  Christian Ledig,et al.  Is the deconvolution layer the same as a convolutional layer? , 2016, ArXiv.

[27]  Johannes R. Sveinsson,et al.  Classification of Pansharpened Urban Satellite Images , 2012, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[28]  Peng Zan,et al.  Locally Linear Detail Injection for Pansharpening , 2017, IEEE Access.

[29]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[30]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Kiyun Yu,et al.  A New Adaptive Component-Substitution-Based Satellite Image Fusion by Using Partial Replacement , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[32]  Andrea Garzelli,et al.  Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis , 2002, IEEE Trans. Geosci. Remote. Sens..

[33]  S. Baronti,et al.  Multispectral and panchromatic data fusion assessment without reference , 2008 .

[34]  Hassan Ghassemian,et al.  A Compressed-Sensing-Based Pan-Sharpening Method for Spectral Distortion Reduction , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[35]  Shutao Li,et al.  Remote Sensing Image Fusion via Sparse Representations Over Learned Dictionaries , 2013, IEEE Transactions on Geoscience and Remote Sensing.

[36]  Pramod K. Varshney,et al.  An Image Fusion Approach Based on Markov Random Fields , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[37]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Yue Huang,et al.  Pan-sharpening with structural consistency and ℓ1/2 gradient prior , 2016 .

[40]  Narendra Ahuja,et al.  Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Te-Ming Tu,et al.  A new look at IHS-like image fusion methods , 2001, Inf. Fusion.

[42]  A. Bovik,et al.  A universal image quality index , 2002, IEEE Signal Processing Letters.

[43]  Lucien Wald,et al.  Data Fusion. Definitions and Architectures - Fusion of Images of Different Spatial Resolutions , 2002 .

[44]  Tae Hyun Kim,et al.  Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  Yaonan Wang,et al.  Using the discrete wavelet frame transform to merge Landsat TM and SPOT panchromatic images , 2002, Inf. Fusion.

[46]  Manjunath V. Joshi,et al.  MAP Estimation for Multiresolution Fusion in Remotely Sensed Images Using an IGMRF Prior Model , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[47]  Hassan Ghassemian,et al.  A new pansharpening method using multi resolution analysis framework and deep neural networks , 2017, 2017 3rd International Conference on Pattern Recognition and Image Analysis (IPRIA).

[48]  Johannes R. Sveinsson,et al.  A New Pansharpening Algorithm Based on Total Variation , 2014, IEEE Geoscience and Remote Sensing Letters.