Pansharpening based on convolutional autoencoder and multi-scale guided filter

In this paper, we propose a pansharpening method based on a convolutional autoencoder. The convolutional autoencoder is a sort of convolutional neural network (CNN) and objective to scale down the input dimension and typify image features with high exactness. First, the autoencoder network is trained to reduce the difference between the degraded panchromatic image patches and reconstruction output original panchromatic image patches. The intensity component, which is developed by adaptive intensity-hue-saturation (AIHS), is then delivered into the trained convolutional autoencoder network to generate an enhanced intensity component of the multi-spectral image. The pansharpening is accomplished by improving the panchromatic image from the enhanced intensity component using a multi-scale guided filter; then, the semantic detail is injected into the upsampled multi-spectral image. Real and degraded datasets are utilized for the experiments, which exhibit that the proposed technique has the ability to preserve the high spatial details and high spectral characteristics simultaneously. Furthermore, experimental results demonstrated that the proposed study performs state-of-the-art results in terms of subjective and objective assessments on remote sensing data.

[1]  Luciano Alparone,et al.  MTF-tailored Multiscale Fusion of High-resolution MS and Pan Imagery , 2006 .

[2]  Yao Zhao,et al.  Cascaded reconstruction network for compressive image sensing , 2018, EURASIP J. Image Video Process..

[3]  Hongyi Liu,et al.  A New Pan-Sharpening Method With Deep Neural Networks , 2015, IEEE Geoscience and Remote Sensing Letters.

[4]  Jocelyn Chanussot,et al.  Indusion: Fusion of Multispectral and Panchromatic Images Using the Induction Scaling Technique , 2008, IEEE Geoscience and Remote Sensing Letters.

[5]  A. Bovik,et al.  A universal image quality index , 2002, IEEE Signal Processing Letters.

[6]  Kiyun Yu,et al.  A New Adaptive Component-Substitution-Based Satellite Image Fusion by Using Partial Replacement , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[7]  J. G. Liu,et al.  Smoothing Filter-based Intensity Modulation: A spectral preserve image fusion technique for improving spatial details , 2000 .

[8]  Wuzhen Shi,et al.  Anchored neighborhood deep network for single-image super-resolution , 2018, EURASIP J. Image Video Process..

[9]  Muazzam Maqsood,et al.  A Deep Siamese Convolution Neural Network for Multi-Class Classification of Alzheimer Disease , 2020, Brain sciences.

[10]  W. J. Carper,et al.  The use of intensity-hue-saturation transformations for merging SPOT panchromatic and multispectral image data , 1990 .

[11]  Shuyuan Yang,et al.  Spatial–Spectral-Graph-Regularized Low-Rank Tensor Decomposition for Multispectral and Hyperspectral Image Fusion , 2018, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[12]  Jian Sun,et al.  Guided Image Filtering , 2010, ECCV.

[13]  Te-Ming Tu,et al.  A new look at IHS-like image fusion methods , 2001, Inf. Fusion.

[14]  S. Baronti,et al.  Multispectral and panchromatic data fusion assessment without reference , 2008 .

[15]  Feng Zhang,et al.  Superpixel guided structure sparsity for multispectral and hyperspectral image fusion over couple dictionary , 2019, Multimedia Tools and Applications.

[16]  Yong Yang,et al.  Multi-frame image super-resolution reconstruction based on spatial information weighted fields of experts , 2020, Multidimens. Syst. Signal Process..

[17]  Andrea Garzelli,et al.  Optimal MMSE Pan Sharpening of Very High Resolution Multispectral Images , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[18]  Jian Sun,et al.  Guided Image Filtering , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Bruno Aiazzi,et al.  Improving Component Substitution Pansharpening Through Multivariate Regression of MS $+$Pan Data , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[20]  Nasser Kehtarnavaz,et al.  Convolutional Autoencoder-Based Multispectral Image Fusion , 2019, IEEE Access.

[21]  Feiniu Yuan,et al.  Remote Sensing Image Fusion Based on Adaptive IHS and Multiscale Guided Filter , 2016, IEEE Access.

[22]  Gang Li,et al.  Remote sensing classification method of vegetation dynamics based on time series Landsat image: a case of opencast mining area in China , 2018, EURASIP J. Image Video Process..

[23]  Hazim Kemal Ekenel,et al.  DermoNet: densely linked convolutional neural network for efficient skin lesion segmentation , 2019, EURASIP J. Image Video Process..

[24]  Yizhou Rao,et al.  A residual convolutional neural network for pan-shaprening , 2017, 2017 International Workshop on Remote Sensing with Intelligent Processing (RSIP).

[25]  Liping Zhang,et al.  Model-based variational pansharpening method with fast generalized intensity–hue–saturation , 2019, Journal of Applied Remote Sensing.

[26]  Alan R. Gillespie,et al.  Color enhancement of highly correlated images. II. Channel ratio and “chromaticity” transformation techniques , 1987 .

[27]  Jocelyn Chanussot,et al.  Comparison of Pansharpening Algorithms: Outcome of the 2006 GRS-S Data-Fusion Contest , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[28]  A. Hegde,et al.  A Review of Quality Metrics for Fused Image , 2015 .

[29]  Michael Möller,et al.  An Adaptive IHS Pan-Sharpening Method , 2010, IEEE Geoscience and Remote Sensing Letters.

[30]  K. Vadrevu,et al.  Spectral angle mapper and object-based classification combined with hyperspectral remote sensing imagery for obtaining land use/cover mapping in a Mediterranean region , 2013 .

[31]  Ahed Abugabah,et al.  Intelligent Information Systems and Image Processing: A Novel Pan-Sharpening Technique Based on Multiscale Decomposition , 2018, ICVIP.

[32]  Jocelyn Chanussot,et al.  A Critical Comparison Among Pansharpening Algorithms , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[33]  Yukihiko Yamashita,et al.  Anisotropic Guided Filtering , 2020, IEEE Transactions on Image Processing.

[34]  Ahmad AL Smadi,et al.  Pansharpening via Deep Guided Filtering Network , 2018 .

[35]  Davide Cozzolino,et al.  Pansharpening by Convolutional Neural Networks , 2016, Remote. Sens..

[36]  Paidamwoyo Mhangara,et al.  Comparison of Image Fusion Techniques Using Satellite Pour l’Observation de la Terre (SPOT) 6 Satellite Imagery , 2020, Applied Sciences.

[37]  P. Chavez,et al.  Extracting spectral contrast in landsat thematic mapper image data using selective principal component analysis , 1989 .

[38]  Davide Cozzolino,et al.  Target-Adaptive CNN-Based Pansharpening , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[39]  J. G. Liu,et al.  Smoothing Filter-based Intensity Modulation : a spectral preserve image fusion technique for improving spatial details , 2001 .

[40]  A. Mookambiga,et al.  Comprehensive review on fusion techniques for spatial information enhancement in hyperspectral imagery , 2016, Multidimens. Syst. Signal Process..

[41]  Johannes R. Sveinsson,et al.  Quantitative Quality Evaluation of Pansharpened Imagery: Consistency Versus Synthesis , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[42]  Serge Dolgikh,et al.  Spontaneous Concept Learning with Deep Autoencoder , 2018, Int. J. Comput. Intell. Syst..

[43]  Luciano Alparone,et al.  Image fusion—the ARSIS concept and some successful implementation schemes , 2003 .

[44]  Wei Lu,et al.  An Adaptive Pansharpening Method by Using Weighted Least Squares Filter , 2016, IEEE Geoscience and Remote Sensing Letters.