A pansharpening scheme using spectral graph wavelet transforms and convolutional neural networks

ABSTRACT The objective of the multispectral pansharpening scheme is to obtain high spatial-spectral resolution multispectral (MS) images using high spectral resolution MS and high spatial resolution panchromatic (Pan) images. Some distortions are found in the multiresolution analysis (MRA) based on pansharpening. It can be minimized by correct matching of the lowpass filter image. This paper illustrates the pansharpening approach that is based on multistage multichannel spectral graph wavelet transform and convolutional neural network (SGWT-PNN). In this scheme, the Pan image is decomposed by a multistage multichannel SGWT, and then a weighted combination of SGWT decomposition produces a lowpass filter component. Using the convolutional neural network model, this lowpass filter image is converted to a better-matched filter image which is completely fit to the MRA-based pansharpening methods. Simulation results in the context of qualitative and quantitative analysis demonstrates the effectiveness of the proposed scheme applied on datasets collected by different satellites.

[1]  Nidhi Saxena,et al.  Pansharpening scheme using filtering in two-dimensional discrete fractional Fourier transform , 2018, IET Image Process..

[2]  Johannes R. Sveinsson,et al.  MTF-Based Deblurring Using a Wiener Filter for CS and MRA Pansharpening Methods , 2016, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[3]  Jocelyn Chanussot,et al.  Full Scale Regression-Based Injection Coefficients for Panchromatic Sharpening , 2018, IEEE Transactions on Image Processing.

[4]  K. K. Sharma,et al.  A hybrid approach for pansharpening using Hilbert vibration decomposition , 2017, 2017 IEEE International Conference on Power, Control, Signals and Instrumentation Engineering (ICPCSI).

[5]  Hongyan Zhang,et al.  A Large-Scale Benchmark Data Set for Evaluating Pansharpening Performance: Overview and Implementation , 2020, IEEE Geoscience and Remote Sensing Magazine.

[6]  Richard Bamler,et al.  A Sparse Image Fusion Algorithm With Application to Pan-Sharpening , 2013, IEEE Transactions on Geoscience and Remote Sensing.

[7]  K. K. Sharma,et al.  Pansharpening approach using Hilbert vibration decomposition , 2017, IET Image Process..

[8]  Michael Möller,et al.  An Adaptive IHS Pan-Sharpening Method , 2010, IEEE Geoscience and Remote Sensing Letters.

[9]  T. Tadono,et al.  Effects of the spectral properties of a panchromatic image on pan-sharpening simulated using hyperspectral data , 2016 .

[10]  L. Alparone,et al.  An MTF-based spectral distortion minimizing model for pan-sharpening of very high resolution multispectral images of urban areas , 2003, 2003 2nd GRSS/ISPRS Joint Workshop on Remote Sensing and Data Fusion over Urban Areas.

[11]  Liangpei Zhang,et al.  Review of the pansharpening methods for remote sensing images based on the idea of meta-analysis: Practical discussion and challenges , 2019, Inf. Fusion.

[12]  S. Baronti,et al.  Multispectral and panchromatic data fusion assessment without reference , 2008 .

[13]  Davide Cozzolino,et al.  Target-Adaptive CNN-Based Pansharpening , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[14]  Nasser Kehtarnavaz,et al.  Convolutional Autoencoder-Based Multispectral Image Fusion , 2019, IEEE Access.

[15]  Chulhee Lee,et al.  Fast and Efficient Panchromatic Sharpening , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[16]  J. Boardman,et al.  Discrimination among semi-arid landscape endmembers using the Spectral Angle Mapper (SAM) algorithm , 1992 .

[17]  Jocelyn Chanussot,et al.  Synthesis of Multispectral Images to High Spatial Resolution: A Critical Review of Fusion Methods Based on Remote Sensing Physics , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[18]  Jocelyn Chanussot,et al.  A Critical Comparison Among Pansharpening Algorithms , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[19]  Bruno Aiazzi,et al.  A Comparison Between Global and Context-Adaptive Pansharpening of Multispectral Images , 2009, IEEE Geoscience and Remote Sensing Letters.

[20]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Yun Zhang,et al.  Understanding image fusion , 2004 .

[22]  Jocelyn Chanussot,et al.  Fusion of Multispectral and Panchromatic Images Based on Morphological Operators , 2016, IEEE Transactions on Image Processing.

[23]  Jun Li,et al.  Pansharpening via Detail Injection Based Convolutional Neural Networks , 2018, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[24]  Lucien Wald,et al.  Data Fusion. Definitions and Architectures - Fusion of Images of Different Spatial Resolutions , 2002 .

[25]  Alan R. Gillespie,et al.  Color enhancement of highly correlated images. II. Channel ratio and “chromaticity” transformation techniques , 1987 .

[26]  Luciano Alparone,et al.  Spatial Methods for Multispectral Pansharpening: Multiresolution Analysis Demystified , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[27]  Rafael García,et al.  Fusion of multispectral and panchromatic images using improved IHS and PCA mergers based on wavelet decomposition , 2004, IEEE Transactions on Geoscience and Remote Sensing.

[28]  Weiwei Sun,et al.  Pansharpening for Cloud-Contaminated Very High-Resolution Remote Sensing Images , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[29]  R. E. Walker,et al.  Color enhancement of highly correlated images. I - Decorrelation and HSI contrast stretches. [hue saturation intensity , 1986 .

[30]  Shutao Li,et al.  Remote Sensing Image Fusion via Sparse Representations Over Learned Dictionaries , 2013, IEEE Transactions on Geoscience and Remote Sensing.

[31]  Kamalesh Kumar Sharma,et al.  Improved facial expression recognition using graph signal processing , 2017 .

[32]  Yuichi Tanaka,et al.  Oversampled Graph Laplacian Matrix for Graph Filter Banks , 2014, IEEE Transactions on Signal Processing.

[33]  Liangpei Zhang,et al.  Boosting the Accuracy of Multispectral Image Pansharpening by Learning a Deep Residual Network , 2017, IEEE Geoscience and Remote Sensing Letters.

[34]  Wojciech Czaja,et al.  Wavelet packet mixing for image fusion and pan-sharpening , 2014, Defense + Security Symposium.

[35]  Hongyi Liu,et al.  A New Pan-Sharpening Method With Deep Neural Networks , 2015, IEEE Geoscience and Remote Sensing Letters.

[36]  A. Bovik,et al.  A universal image quality index , 2002, IEEE Signal Processing Letters.

[37]  Shutao Li,et al.  A New Pan-Sharpening Method Using a Compressed Sensing Technique , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[38]  Sunil K. Narang,et al.  Perfect Reconstruction Two-Channel Wavelet Filter Banks for Graph Structured Data , 2011, IEEE Transactions on Signal Processing.

[39]  Luciano Alparone,et al.  MTF-tailored Multiscale Fusion of High-resolution MS and Pan Imagery , 2006 .

[40]  Shutao Li,et al.  Remote Sensing Image Fusion Method Based on Nonsubsampled Shearlet Transform and Sparse Representation , 2015, Sensing and Imaging.

[41]  Xavier Otazu,et al.  Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods , 2005, IEEE Transactions on Geoscience and Remote Sensing.

[42]  S. Sides,et al.  Comparison of three different methods to merge multiresolution and multispectral data: Landsat TM and SPOT panchromatic , 1991 .

[43]  Luciano Alparone,et al.  Bi-cubic interpolation for shift-free pan-sharpening , 2013 .

[44]  Pierre Vandergheynst,et al.  Wavelets on Graphs via Spectral Graph Theory , 2009, ArXiv.

[45]  Davide Cozzolino,et al.  Pansharpening by Convolutional Neural Networks , 2016, Remote. Sens..