A New Benchmark Based on Recent Advances in Multispectral Pansharpening: Revisiting Pansharpening With Classical and Emerging Pansharpening Methods

Pansharpening refers to the fusion of a multispectral (MS) image and panchromatic (PAN) data aimed at generating an outcome with the same spatial resolution of the PAN data and the spectral resolution of the MS image. In the last 30 years, several approaches to deal with this issue have been proposed. However, the reproducibility of these methods is often limited, making the comparison with the state of the art hard to achieve. Thus, to fill this gap, we propose a new benchmark consisting of recent advances in MS pansharpening. In particular, optimized classical approaches [multiresolution analysis (MRA) and component substitution (CS)] are compared with methods belonging to the third generation of pansharpening, represented by variational optimization-based (VO) and machine learning (ML) techniques. The benchmark is tested on different scenarios (from urban to rural areas) acquired by different commercial sensors [i.e., IKONOS (IK), GeoEye-1 (GE-1), and WorldView-3 (WV-3)]. Both quantitative and qualitative assessments and the computational burden are analyzed in this article, and all of the implementations have been collected in a MATLAB toolbox that is made available to the community.

[1]  Hairong Qi,et al.  Unsupervised and Unregistered Hyperspectral Image Super-Resolution With Mutual Dirichlet-Net , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[2]  Chen Chen,et al.  Pan-GAN: An unsupervised pan-sharpening method for remote sensing image fusion , 2020, Inf. Fusion.

[3]  Jocelyn Chanussot,et al.  Fusion of short-wave infrared and visible near-infrared WorldView-3 data , 2020, Inf. Fusion.

[4]  Johannes R. Sveinsson,et al.  Model-Based Reduced-Rank Pansharpening , 2020, IEEE Geoscience and Remote Sensing Letters.

[5]  Giuseppe Scarpa,et al.  A Detail-Preserving Cross-Scale Learning Strategy for CNN-Based Pansharpening , 2020, Remote. Sens..

[6]  Jocelyn Chanussot,et al.  A Pansharpening Approach Based on Multiple Linear Regression Estimation of Injection Coefficients , 2020, IEEE Geoscience and Remote Sensing Letters.

[7]  Giuseppe Scarpa,et al.  Fast Super-Resolution of 20 m Sentinel-2 Bands Using Convolutional Neural Networks , 2019, Remote. Sens..

[8]  Luciano Alparone,et al.  Fast Reproducible Pansharpening Based on Instrument and Acquisition Modeling: AWLP Revisited , 2019, Remote. Sens..

[9]  José M. Bioucas-Dias,et al.  Panchromatic Sharpening of Multispectral Satellite Imagery Via an Explicitly Defined Convex Self-Similarity Regularization , 2019, IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium.

[10]  Jocelyn Chanussot,et al.  Assessment of Hyperspectral Sharpening Methods for the Monitoring of Natural Areas Using Multiplatform Remote Sensing Imagery , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[11]  Johannes R. Sveinsson,et al.  Sentinel-2 Sharpening Using a Reduced-Rank Method , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[12]  Gemine Vivone,et al.  Robust Band-Dependent Spatial-Detail Approaches for Panchromatic Sharpening , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[13]  Mingwei Sun,et al.  Pan-Sharpening Using an Efficient Bidirectional Pyramid Network , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[14]  Jocelyn Chanussot,et al.  A Combiner-Based Full Resolution Quality Assessment Index for Pansharpening , 2019, IEEE Geoscience and Remote Sensing Letters.

[15]  Liangpei Zhang,et al.  Review of the pansharpening methods for remote sensing images based on the idea of meta-analysis: Practical discussion and challenges , 2019, Inf. Fusion.

[16]  Haitao Yin PAN-Guided Cross-Resolution Projection for Local Adaptive Sparse Representation- Based Pansharpening , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[17]  Jocelyn Chanussot,et al.  Pansharpening Based on Deconvolution for Multiband Filter Estimation , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[18]  Jon Atli Benediktsson,et al.  Multisource and Multitemporal Data Fusion in Remote Sensing , 2018, ArXiv.

[19]  Fang Li,et al.  High-Quality Bayesian Pansharpening , 2019, IEEE Transactions on Image Processing.

[20]  Agudelo Medina,et al.  Perceptual Quality Assessment of Pan-sharpened Images , 2018 .

[21]  Huiming Tang,et al.  Pixel-wise regression using U-Net and its application on pansharpening , 2018, Neurocomputing.

[22]  Jocelyn Chanussot,et al.  A Bayesian Procedure for Full-Resolution Quality Assessment of Pansharpened Products , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[23]  Luciano Alparone,et al.  Multispectral Pansharpening with Radiative Transfer-Based Detail-Injection Modeling for Preserving Changes in Vegetation Cover , 2018, Remote. Sens..

[24]  Xiangyu Liu,et al.  Psgan: A Generative Adversarial Network for Remote Sensing Image Pan-Sharpening , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[25]  Hairong Qi,et al.  Unsupervised Sparse Dirichlet-Net for Hyperspectral Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[26]  Jocelyn Chanussot,et al.  Full Scale Regression-Based Injection Coefficients for Panchromatic Sharpening , 2018, IEEE Transactions on Image Processing.

[27]  Zhenfeng Shao,et al.  Remote Sensing Image Fusion With Deep Convolutional Neural Network , 2018, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[28]  Konrad Schindler,et al.  Super-Resolution of Sentinel-2 Images: Learning a Globally Applicable Deep Neural Network , 2018, ISPRS Journal of Photogrammetry and Remote Sensing.

[29]  Tao Li,et al.  A Variational Pan-Sharpening Method Based on Spatial Fractional-Order Geometry and Spectral–Spatial Low-Rank Priors , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[30]  Jocelyn Chanussot,et al.  A Regression-Based High-Pass Modulation Pansharpening Approach , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[31]  Liangpei Zhang,et al.  A Multiscale and Multidepth Convolutional Neural Network for Remote Sensing Imagery Pan-Sharpening , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[32]  Xiao Xiang Zhu,et al.  Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources , 2017, IEEE Geoscience and Remote Sensing Magazine.

[33]  Qian Du,et al.  Hyperspectral Image Spatial Super-Resolution via 3D Full Convolutional Neural Network , 2017, Remote. Sens..

[34]  Luciano Alparone,et al.  Haze Correction for Contrast-Based Multispectral Pansharpening , 2017, IEEE Geoscience and Remote Sensing Letters.

[35]  Junfeng Yang,et al.  PanNet: A Deep Network Architecture for Pan-Sharpening , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[36]  Jocelyn Chanussot,et al.  A variational pansharpening approach based on reproducible kernel Hilbert space and heaviside function , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[37]  Chiman Kwan,et al.  Blind Quality Assessment of Fused WorldView-3 Images by Using the Combinations of Pansharpening and Hypersharpening Paradigms , 2017, IEEE Geoscience and Remote Sensing Letters.

[38]  Hairong Qi,et al.  DOES multispectral / hyperspectral pansharpening improve the performance of anomaly detection? , 2017, 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[39]  Liangpei Zhang,et al.  Boosting the Accuracy of Multispectral Image Pansharpening by Learning a Deep Residual Network , 2017, IEEE Geoscience and Remote Sensing Letters.

[40]  Yizhou Rao,et al.  A residual convolutional neural network for pan-shaprening , 2017, 2017 International Workshop on Remote Sensing with Intelligent Processing (RSIP).

[41]  Jocelyn Chanussot,et al.  Band Assignment Approaches for Hyperspectral Sharpening , 2017, IEEE Geoscience and Remote Sensing Letters.

[42]  Johannes R. Sveinsson,et al.  Multispectral and Hyperspectral Image Fusion Using a 3-D-Convolutional Neural Network , 2017, IEEE Geoscience and Remote Sensing Letters.

[43]  Jaco Kemp,et al.  Effect of pan-sharpening multi-temporal Landsat 8 imagery for crop type differentiation using different classification techniques , 2017, Comput. Electron. Agric..

[44]  Jocelyn Chanussot,et al.  Context-Adaptive Pansharpening Based on Image Segmentation , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[45]  Shutao Li,et al.  Pixel-level image fusion: A survey of the state of the art , 2017, Inf. Fusion.

[46]  Luciano Alparone,et al.  Sensitivity of Pansharpening Methods to Temporal and Instrumental Changes Between Multispectral and Panchromatic Data Sets , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[47]  Xiao Xiang Zhu,et al.  Data Fusion and Remote Sensing: An ever-growing relationship , 2016, IEEE Geoscience and Remote Sensing Magazine.

[48]  Hassan Ghassemian,et al.  A review of remote sensing image fusion methods , 2016, Inf. Fusion.

[49]  Andrea Garzelli,et al.  A Review of Image Fusion Algorithms Based on the Super-Resolution Paradigm , 2016, Remote. Sens..

[50]  Hassan Ghassemian,et al.  Nonlinear IHS: A Promising Method for Pan-Sharpening , 2016, IEEE Geoscience and Remote Sensing Letters.

[51]  Davide Cozzolino,et al.  Pansharpening by Convolutional Neural Networks , 2016, Remote. Sens..

[52]  Gwendoline Blanchet,et al.  A Survey of Pansharpening Methods with A New Band-Decoupled Variational Model , 2016, ArXiv.

[53]  Jinying Zhong,et al.  Remote Sensing Image Fusion with Convolutional Neural Network , 2016 .

[54]  Richard Bamler,et al.  Exploiting Joint Sparsity for Pansharpening: The J-SparseFI Algorithm , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[55]  Johannes R. Sveinsson,et al.  MTF-Based Deblurring Using a Wiener Filter for CS and MRA Pansharpening Methods , 2016, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[56]  Jocelyn Chanussot,et al.  Fusion of Multispectral and Panchromatic Images Based on Morphological Operators , 2016, IEEE Transactions on Image Processing.

[57]  Shunlin Liang,et al.  Pan-sharpening using a guided filter , 2016 .

[58]  Johannes R. Sveinsson,et al.  Quantitative Quality Evaluation of Pansharpened Imagery: Consistency Versus Synthesis , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[59]  Luciano Alparone,et al.  Spatial Methods for Multispectral Pansharpening: Multiresolution Analysis Demystified , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[60]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[61]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[62]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[63]  Bruno Aiazzi,et al.  Full-Scale Assessment of Pansharpening Through Polynomial Fitting of Multiscale Measurements , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[64]  Bruno Aiazzi,et al.  Hyper-Sharpening: A First Approach on SIM-GA Data , 2015, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[65]  Aleksandra Pizurica,et al.  Two-stage fusion of thermal hyperspectral and visible RGB image by PCA and guided filter , 2015, 2015 7th Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS).

[66]  Jocelyn Chanussot,et al.  A Critical Comparison Among Pansharpening Algorithms , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[67]  Johannes R. Sveinsson,et al.  Model-Based Fusion of Multi- and Hyperspectral Images Using PCA and Wavelets , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[68]  Naoto Yokoya,et al.  Hyperspectral Pansharpening: A Review , 2015, IEEE Geoscience and Remote Sensing Magazine.

[69]  Jocelyn Chanussot,et al.  Pansharpening Based on Semiblind Deconvolution , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[70]  Andrea Garzelli,et al.  Pansharpening of Multispectral Images Based on Nonlocal Parameter Optimization , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[71]  S. Baronti,et al.  Remote Sensing Image Fusion , 2015 .

[72]  Aimin Zhou,et al.  Pan-sharpening of multi-spectral images using a new variational model , 2015 .

[73]  Hassan Ghassemian,et al.  Fusion of MS and PAN Images Preserving Spectral Quality , 2015, IEEE Geoscience and Remote Sensing Letters.

[74]  Jocelyn Chanussot,et al.  A Pansharpening Method Based on the Sparse Representation of Injected Details , 2015, IEEE Geoscience and Remote Sensing Letters.

[75]  Luciano Alparone,et al.  Full scale assessment of pansharpening methods and data products , 2014, Remote Sensing.

[76]  Jon Atli Benediktsson,et al.  Pansharpening With Matting Model , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[77]  Jocelyn Chanussot,et al.  MultiResolution Analysis and Component Substitution techniques for hyperspectral Pansharpening , 2014, 2014 IEEE Geoscience and Remote Sensing Symposium.

[78]  Jocelyn Chanussot,et al.  A New Pansharpening Method Based on Spatial and Spectral Sparsity Priors , 2014, IEEE Transactions on Image Processing.

[79]  Gaurav Sharma,et al.  A Regularized Model-Based Optimization Framework for Pan-Sharpening , 2014, IEEE Transactions on Image Processing.

[80]  Jocelyn Chanussot,et al.  Contrast and Error-Based Fusion Schemes for Multispectral Image Pansharpening , 2014, IEEE Geoscience and Remote Sensing Letters.

[81]  Bo Li,et al.  High-Fidelity Component Substitution Pansharpening by the Fitting of Substitution Data , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[82]  Bamdev Mishra,et al.  Manopt, a matlab toolbox for optimization on manifolds , 2013, J. Mach. Learn. Res..

[83]  Johannes R. Sveinsson,et al.  A New Pansharpening Algorithm Based on Total Variation , 2014, IEEE Geoscience and Remote Sensing Letters.

[84]  Ming Cheng,et al.  Sparse Representation Based Pansharpening Using Trained Dictionary , 2014, IEEE Geoscience and Remote Sensing Letters.

[85]  Richard Bamler,et al.  A Sparse Image Fusion Algorithm With Application to Pan-Sharpening , 2013, IEEE Transactions on Geoscience and Remote Sensing.

[86]  Shutao Li,et al.  Remote Sensing Image Fusion via Sparse Representations Over Learned Dictionaries , 2013, IEEE Transactions on Geoscience and Remote Sensing.

[87]  Peijun Du,et al.  Information fusion techniques for change detection from multi-temporal remote sensing images , 2013, Inf. Fusion.

[88]  Luciano Alparone,et al.  Advantages of Laplacian pyramids over ''à trous'' wavelet transforms for pansharpening of multispectral images , 2012, Remote Sensing.

[89]  Liangpei Zhang,et al.  A Practical Compressed Sensing-Based Pan-Sharpening Method , 2012, IEEE Geoscience and Remote Sensing Letters.

[90]  Yifan Zhang,et al.  A Bayesian Restoration Approach for Hyperspectral Images , 2012, IEEE Transactions on Geoscience and Remote Sensing.

[91]  S. Baronti,et al.  Twenty-Five Years of Pansharpening: A Critical Review and New Developments , 2012 .

[92]  Michael Möller,et al.  A Variational Approach for Sharpening High Dimensional Images , 2012, SIAM J. Imaging Sci..

[93]  Jocelyn Chanussot,et al.  Fusion of hyperspectral and panchromatic images using multiresolution analysis and nonlinear PCA band reduction , 2011, 2011 IEEE International Geoscience and Remote Sensing Symposium.

[94]  Shutao Li,et al.  A New Pan-Sharpening Method Using a Compressed Sensing Technique , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[95]  Kiyun Yu,et al.  A New Adaptive Component-Substitution-Based Satellite Image Fusion by Using Partial Replacement , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[96]  Jocelyn Chanussot,et al.  Pansharpening Quality Assessment Using the Modulation Transfer Functions of Instruments , 2009, IEEE Transactions on Geoscience and Remote Sensing.

[97]  Andrea Garzelli,et al.  Hypercomplex Quality Assessment of Multi/Hyperspectral Images , 2009, IEEE Geoscience and Remote Sensing Letters.

[98]  Yifan Zhang,et al.  Noise-Resistant Wavelet-Based Bayesian Fusion of Multispectral and Hyperspectral Images , 2009, IEEE Transactions on Geoscience and Remote Sensing.

[99]  Levent Tunçel,et al.  Optimization algorithms on matrix manifolds , 2009, Math. Comput..

[100]  Julien Radoux,et al.  Bayesian Data Fusion for Adaptable Image Pansharpening , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[101]  Roger L. King,et al.  An Efficient Pan-Sharpening Method via a Combined Adaptive PCA Approach and Contourlets , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[102]  Wenzhong Shi,et al.  Remote Sensing Image Fusion Using Multiscale Mapped LS-SVM , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[103]  Jocelyn Chanussot,et al.  Synthesis of Multispectral Images to High Spatial Resolution: A Critical Review of Fusion Methods Based on Remote Sensing Physics , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[104]  S. Baronti,et al.  Multispectral and panchromatic data fusion assessment without reference , 2008 .

[105]  Andrea Garzelli,et al.  Optimal MMSE Pan Sharpening of Very High Resolution Multispectral Images , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[106]  Bruno Aiazzi,et al.  Improving Component Substitution Pansharpening Through Multivariate Regression of MS $+$Pan Data , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[107]  Jocelyn Chanussot,et al.  Comparison of Pansharpening Algorithms: Outcome of the 2006 GRS-S Data-Fusion Contest , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[108]  Luciano Alparone,et al.  Remote sensing image fusion using the curvelet transform , 2007, Inf. Fusion.

[109]  Laura Igual,et al.  A Variational Model for P+XS Image Fusion , 2006, International Journal of Computer Vision.

[110]  Luciano Alparone,et al.  MTF-tailored Multiscale Fusion of High-resolution MS and Pan Imagery , 2006 .

[111]  Minh N. Do,et al.  Ieee Transactions on Image Processing the Contourlet Transform: an Efficient Directional Multiresolution Image Representation , 2022 .

[112]  Xavier Otazu,et al.  Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods , 2005, IEEE Transactions on Geoscience and Remote Sensing.

[113]  Xavier Otazu,et al.  Comparison between Mallat's and the ‘à trous’ discrete wavelet transform based algorithms for the fusion of multispectral and panchromatic images , 2005 .

[114]  Luciano Alparone,et al.  A global quality measurement of pan-sharpened multispectral imagery , 2004, IEEE Geoscience and Remote Sensing Letters.

[115]  Te-Ming Tu,et al.  A fast intensity-hue-saturation fusion technique with spectral adjustment for IKONOS imagery , 2004, IEEE Geoscience and Remote Sensing Letters.

[116]  Luciano Alparone,et al.  Image fusion—the ARSIS concept and some successful implementation schemes , 2003 .

[117]  Andrea Garzelli,et al.  Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis , 2002, IEEE Trans. Geosci. Remote. Sens..

[118]  A. Bovik,et al.  A universal image quality index , 2002, IEEE Signal Processing Letters.

[119]  Lucien Wald,et al.  Data Fusion. Definitions and Architectures - Fusion of Images of Different Spatial Resolutions , 2002 .

[120]  Emmanuel J. Candès,et al.  The curvelet transform for image denoising , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[121]  Te-Ming Tu,et al.  A new look at IHS-like image fusion methods , 2001, Inf. Fusion.

[122]  J. G. Liu,et al.  Smoothing Filter-based Intensity Modulation : a spectral preserve image fusion technique for improving spatial details , 2001 .

[123]  Xavier Otazu,et al.  Multiresolution-based image fusion with additive wavelet decomposition , 1999, IEEE Trans. Geosci. Remote. Sens..

[124]  Y. Zhang,et al.  A new merging method and its spectral and spatial effects , 1999 .

[125]  J. Zhou,et al.  A wavelet transform method to merge Landsat TM and SPOT panchromatic data , 1998 .

[126]  Robert A. Schowengerdt,et al.  Remote sensing, models, and methods for image processing , 1997 .

[127]  L. Wald,et al.  Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images , 1997 .

[128]  Jelena Kovacevic,et al.  Wavelets and Subband Coding , 2013, Prentice Hall Signal Processing Series.

[129]  B. Silverman,et al.  The Stationary Wavelet Transform and some Statistical Applications , 1995 .

[130]  J. Boardman,et al.  Discrimination among semi-arid landscape endmembers using the Spectral Angle Mapper (SAM) algorithm , 1992 .

[131]  V. K. Shettigara,et al.  A generalized component substitution technique for spatial enhancement of multispectral images using , 1992 .

[132]  S. Sides,et al.  Comparison of three different methods to merge multiresolution and multispectral data: Landsat TM and SPOT panchromatic , 1991 .

[133]  W. J. Carper,et al.  The use of intensity-hue-saturation transformations for merging SPOT panchromatic and multispectral image data , 1990 .

[134]  Stéphane Mallat,et al.  A Theory for Multiresolution Signal Decomposition: The Wavelet Representation , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[135]  P. Chavez,et al.  Extracting spectral contrast in landsat thematic mapper image data using selective principal component analysis , 1989 .

[136]  Alan R. Gillespie,et al.  Color enhancement of highly correlated images. II. Channel ratio and “chromaticity” transformation techniques , 1987 .

[137]  Edward H. Adelson,et al.  The Laplacian Pyramid as a Compact Image Code , 1983, IEEE Trans. Commun..