Convolutional Autoencoder-Based Multispectral Image Fusion

This paper presents a deep learning-based pansharpening method for fusion of panchromatic and multispectral images in remote sensing applications. This method can be categorized as a component substitution method in which a convolutional autoencoder network is trained to generate original panchromatic images from their spatially degraded versions. Low resolution multispectral images are then fed into the trained convolutional autoencoder network to generate estimated high resolution multispectral images. The fusion is achieved by injecting the detail map of each spectral band into the corresponding estimated high resolution multispectral bands. Full reference and no-reference metrics are computed for the images of three satellite datasets. These measures are compared with the existing fusion methods whose codes are publicly available. The results obtained indicate the effectiveness of the developed deep learning-based method for multispectral image fusion.

[1]  Giuseppe Scarpa,et al.  A CNN-Based Fusion Method for Super-Resolution of Sentinel-2 Data , 2018, IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium.

[2]  Fang Liu,et al.  Coupled Tensor Decomposition for Hyperspectral Pansharpening , 2018, IEEE Access.

[3]  Hassan Ghassemian,et al.  An adaptive multispectral image fusion using particle swarm optimization , 2017, 2017 Iranian Conference on Electrical Engineering (ICEE).

[4]  Wei Lu,et al.  An Adaptive Pansharpening Method by Using Weighted Least Squares Filter , 2016, IEEE Geoscience and Remote Sensing Letters.

[5]  Xiangyu Liu,et al.  Psgan: A Generative Adversarial Network for Remote Sensing Image Pan-Sharpening , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[6]  Johannes R. Sveinsson,et al.  Model based pansharpening method based on TV and MTF deblurring , 2015, 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[7]  Giampaolo Ferraioli,et al.  A CNN-Based Model for Pansharpening of WorldView-3 Images , 2018, IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium.

[8]  Luciano Alparone,et al.  MTF-tailored Multiscale Fusion of High-resolution MS and Pan Imagery , 2006 .

[9]  Jon Atli Benediktsson,et al.  Pansharpening With Matting Model , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[10]  Jocelyn Chanussot,et al.  A Pansharpening Method Based on the Sparse Representation of Injected Details , 2015, IEEE Geoscience and Remote Sensing Letters.

[11]  Yizhou Rao,et al.  A residual convolutional neural network for pan-shaprening , 2017, 2017 International Workshop on Remote Sensing with Intelligent Processing (RSIP).

[12]  Hassan Ghassemian,et al.  A review of remote sensing image fusion methods , 2016, Inf. Fusion.

[13]  Hongyi Liu,et al.  A New Pan-Sharpening Method With Deep Neural Networks , 2015, IEEE Geoscience and Remote Sensing Letters.

[14]  Davide Cozzolino,et al.  Target-Adaptive CNN-Based Pansharpening , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[15]  Hassan Ghassemian,et al.  Incorporating an Adaptive Image Prior Model Into Bayesian Fusion of Multispectral and Panchromatic Images , 2018, IEEE Geoscience and Remote Sensing Letters.

[16]  Hassan Ghassemian,et al.  A new pansharpening method using multi resolution analysis framework and deep neural networks , 2017, 2017 3rd International Conference on Pattern Recognition and Image Analysis (IPRIA).

[17]  Hassan Ghassemian,et al.  Combining the spectral PCA and spatial PCA fusion methods by an optimal filter , 2016, Inf. Fusion.

[18]  Xu Li,et al.  Pansharpening Based on Joint Gaussian Guided Upsampling , 2018, IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium.

[19]  H. Ghassemian,et al.  A multi-objective component-substitution-based pansharpening , 2017, 2017 3rd International Conference on Pattern Recognition and Image Analysis (IPRIA).

[20]  Johannes R. Sveinsson,et al.  Model-Based Satellite Image Fusion , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[21]  Hassan Ghassemian,et al.  Application of fractional-order differentiation in multispectral image fusion , 2018 .

[22]  Hassan Ghassemian,et al.  Panchromatic and multispectral images fusion using sparse representation , 2017, 2017 Artificial Intelligence and Signal Processing Conference (AISP).

[23]  Kiyun Yu,et al.  A New Adaptive Component-Substitution-Based Satellite Image Fusion by Using Partial Replacement , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[24]  Ye Zhang,et al.  An improved non-subsampled contourlet transform-based hybrid pan-sharpening algorithm , 2017, 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[25]  Hassan Ghassemian,et al.  Nonlinear IHS: A Promising Method for Pan-Sharpening , 2016, IEEE Geoscience and Remote Sensing Letters.

[26]  J. Zhou,et al.  A wavelet transform method to merge Landsat TM and SPOT panchromatic data , 1998 .

[27]  Shuyuan Yang,et al.  Pansharpening With Multiscale Geometric Support Tensor Machine , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[28]  Lucien Wald,et al.  Data Fusion. Definitions and Architectures - Fusion of Images of Different Spatial Resolutions , 2002 .

[29]  Hui Li,et al.  Improvement of MRA-Based Pansharpening Methods Through the Considerasion of Mixed Pixels , 2018, IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium.

[30]  Sylvie Le Hégarat-Mascle,et al.  High spectral quality pansharpening approach based on MTF-matched filter banks , 2016, Multidimens. Syst. Signal Process..

[31]  Jocelyn Chanussot,et al.  Indusion: Fusion of Multispectral and Panchromatic Images Using the Induction Scaling Technique , 2008, IEEE Geoscience and Remote Sensing Letters.

[32]  A. Bovik,et al.  A universal image quality index , 2002, IEEE Signal Processing Letters.

[33]  Lei Wu,et al.  Remote Sensing Image Fusion Based on Adaptively Weighted Joint Detail Injection , 2018, IEEE Access.

[34]  Michael Möller,et al.  An Adaptive IHS Pan-Sharpening Method , 2010, IEEE Geoscience and Remote Sensing Letters.

[35]  Rabab Kreidieh Ward,et al.  Deep learning for pixel-level image fusion: Recent advances and future prospects , 2018, Inf. Fusion.

[36]  Jocelyn Chanussot,et al.  A Critical Comparison Among Pansharpening Algorithms , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[37]  Fang Li,et al.  High-Quality Bayesian Pansharpening , 2019, IEEE Transactions on Image Processing.

[38]  Te-Ming Tu,et al.  Generalized IHS-BT framework for the pansharpening of high-resolution satellite imagery , 2018, Journal of Applied Remote Sensing.

[39]  Tao Li,et al.  A Variational Pan-Sharpening Method Based on Spatial Fractional-Order Geometry and Spectral–Spatial Low-Rank Priors , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[40]  Jiangshe Zhang,et al.  An Improved Adaptive Intensity–Hue–Saturation Method for the Fusion of Remote Sensing Images , 2014, IEEE Geoscience and Remote Sensing Letters.

[41]  Lei Wu,et al.  Compensation Details-Based Injection Model for Remote Sensing Image Fusion , 2018, IEEE Geoscience and Remote Sensing Letters.

[42]  Hamid Reza Shahdoosti,et al.  Pansharpening of Clustered MS and Pan Images Considering Mixed Pixels , 2017, IEEE Geoscience and Remote Sensing Letters.

[43]  Liangpei Zhang,et al.  Boosting the Accuracy of Multispectral Image Pansharpening by Learning a Deep Residual Network , 2017, IEEE Geoscience and Remote Sensing Letters.