A review of image fusion techniques for pan-sharpening of high-resolution satellite imagery

Abstract Pan-sharpening methods are commonly used to synthesize multispectral and panchromatic images. Selecting an appropriate algorithm that maintains the spectral and spatial information content of input images is a challenging task. This review paper investigates a wide range of algorithms, including 41 methods. For this purpose, the methods were categorized as Component Substitution (CS-based), Multi-Resolution Analysis (MRA), Variational Optimization-based (VO), and Hybrid and were tested on a collection of 21 case studies. These include images from WorldView-2, 3 & 4, GeoEye-1, QuickBird, IKONOS, KompSat-2, KompSat-3A, TripleSat, Pleiades-1, Pleiades with the aerial platform, and Deimos-2. Neural network-based methods were excluded due to their substantial computational requirements for operational mapping purposes. The methods were evaluated based on four Spectral and three Spatial quality metrics. An Analysis Of Variance (ANOVA) was used to statistically compare the pan-sharpening categories. Results indicate that MRA-based methods performed better in terms of spectral quality, whereas most Hybrid-based methods had the highest spatial quality and CS-based methods had the lowest results both spectrally and spatially. The revisited version of the Additive Wavelet Luminance Proportional Pan-sharpening method had the highest spectral quality, whereas Generalized IHS with Best Trade-off Parameter with Additive Weights showed the highest spatial quality. CS-based methods generally had the fastest run-time, whereas the majority of methods belonging to MRA and VO categories had relatively long run times.

[1]  Lei Zhang,et al.  A sparse representation based pansharpening method , 2018, Future Gener. Comput. Syst..

[2]  Richard Bamler,et al.  A Sparse Image Fusion Algorithm With Application to Pan-Sharpening , 2013, IEEE Transactions on Geoscience and Remote Sensing.

[3]  Chulhee Lee,et al.  Fast and Efficient Panchromatic Sharpening , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[4]  Wl Lau,et al.  Comparison of image data fusion techniques using entropy and INI , 2016 .

[5]  Jocelyn Chanussot,et al.  Full Scale Regression-Based Injection Coefficients for Panchromatic Sharpening , 2018, IEEE Transactions on Image Processing.

[6]  Aggelos K. Katsaggelos,et al.  A survey of classical methods and new trends in pansharpening of multispectral images , 2011, EURASIP J. Adv. Signal Process..

[7]  Mingwei Sun,et al.  Pan-Sharpening Using an Efficient Bidirectional Pyramid Network , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[9]  Gemine Vivone,et al.  Robust Band-Dependent Spatial-Detail Approaches for Panchromatic Sharpening , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[10]  Haitao Yin,et al.  A Joint Sparse and Low-Rank Decomposition for Pansharpening of Multispectral Images , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[11]  Andrea Garzelli,et al.  Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis , 2002, IEEE Trans. Geosci. Remote. Sens..

[12]  Hongyi Liu,et al.  A New Pan-Sharpening Method With Deep Neural Networks , 2015, IEEE Geoscience and Remote Sensing Letters.

[13]  Julien Radoux,et al.  Bayesian Data Fusion for Adaptable Image Pansharpening , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[14]  J. Zhou,et al.  A wavelet transform method to merge Landsat TM and SPOT panchromatic data , 1998 .

[15]  Luciano Alparone,et al.  Intersensor Statistical Matching for Pansharpening: Theoretical Issues and Practical Solutions , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[16]  Vladimir Petrovic,et al.  Objective image fusion performance measure , 2000 .

[17]  Nasser Kehtarnavaz,et al.  Convolutional Autoencoder-Based Multispectral Image Fusion , 2019, IEEE Access.

[18]  Jocelyn Chanussot,et al.  Multi-resolution analysis techniques and nonlinear PCA for hybrid pansharpening applications , 2016, Multidimens. Syst. Signal Process..

[19]  Hassan Ghassemian,et al.  Nonlinear IHS: A Promising Method for Pan-Sharpening , 2016, IEEE Geoscience and Remote Sensing Letters.

[20]  Jocelyn Chanussot,et al.  Fusion of Multispectral and Panchromatic Images Based on Morphological Operators , 2016, IEEE Transactions on Image Processing.

[21]  A. AndersonJ. Pansharpening of Multispectral Satellite Images via Lattice Structures , 2016 .

[22]  Xavier Otazu,et al.  Multiresolution-based image fusion with additive wavelet decomposition , 1999, IEEE Trans. Geosci. Remote. Sens..

[23]  Bo Huang,et al.  Super-Resolution-Guided Progressive Pansharpening Based on a Deep Convolutional Neural Network , 2020, IEEE Transactions on Geoscience and Remote Sensing.

[24]  A. Hegde,et al.  A Review of Quality Metrics for Fused Image , 2015 .

[25]  Hairong Qi,et al.  Unsupervised Pansharpening Based on Self-Attention Mechanism , 2020, IEEE Transactions on Geoscience and Remote Sensing.

[26]  Xiangyu Liu,et al.  Remote sensing image fusion based on two-stream fusion network , 2020, Inf. Fusion.

[27]  Flavio Esposito,et al.  UAV-Based High Resolution Thermal Imaging for Vegetation Monitoring, and Plant Phenotyping Using ICI 8640 P, FLIR Vue Pro R 640, and thermoMap Cameras , 2019, Remote. Sens..

[28]  Jocelyn Chanussot,et al.  Indusion: Fusion of Multispectral and Panchromatic Images Using the Induction Scaling Technique , 2008, IEEE Geoscience and Remote Sensing Letters.

[29]  Yu Liu,et al.  A practical pan-sharpening method with wavelet transform and sparse representation , 2013, 2013 IEEE International Conference on Imaging Systems and Techniques (IST).

[30]  A. Bovik,et al.  A universal image quality index , 2002, IEEE Signal Processing Letters.

[31]  K. Nikolakopoulos Comparison of Nine Fusion Techniques for Very High Resolution Data , 2008 .

[32]  Kiyun Yu,et al.  A New Adaptive Component-Substitution-Based Satellite Image Fusion by Using Partial Replacement , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[33]  Laura Igual,et al.  A Variational Model for P+XS Image Fusion , 2006, International Journal of Computer Vision.

[34]  Linhai Jing,et al.  Improvement of a Pansharpening Method Taking Into Account Haze , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[35]  Alfred Stein,et al.  Spatiotemporal Image Fusion in Remote Sensing , 2019, Remote. Sens..

[36]  Yue Huang,et al.  Pan-sharpening with structural consistency and ℓ1/2 gradient prior , 2016 .

[37]  T. H. Meyer,et al.  Evaluation of pansharpening algorithms in support of earth observation based rapid-mapping workflows , 2013 .

[38]  Jocelyn Chanussot,et al.  A Critical Comparison Among Pansharpening Algorithms , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[39]  Jocelyn Chanussot,et al.  Pansharpening Quality Assessment Using the Modulation Transfer Functions of Instruments , 2009, IEEE Transactions on Geoscience and Remote Sensing.

[40]  Andrea Garzelli,et al.  Optimal MMSE Pan Sharpening of Very High Resolution Multispectral Images , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[41]  Bruno Aiazzi,et al.  Improving Component Substitution Pansharpening Through Multivariate Regression of MS $+$Pan Data , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[42]  Gh. S. El-Tawel,et al.  An integrated scheme to improve pan-sharpening visual quality of satellite images , 2015 .

[43]  Jocelyn Chanussot,et al.  Pansharpening Based on Semiblind Deconvolution , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[44]  Jie Li,et al.  A differential information residual convolutional neural network for pansharpening , 2020, ISPRS Journal of Photogrammetry and Remote Sensing.

[45]  Pan Lin,et al.  A Novel Pan-Sharpening Framework Based on Matting Model and Multiscale Transform , 2017, Remote. Sens..

[46]  Luciano Alparone,et al.  MTF-tailored Multiscale Fusion of High-resolution MS and Pan Imagery , 2006 .

[47]  Hassan Ghassemian,et al.  A review of remote sensing image fusion methods , 2016, Inf. Fusion.

[48]  Richard Bamler,et al.  Exploiting Joint Sparsity for Pansharpening: The J-SparseFI Algorithm , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[49]  G. Rohith,et al.  Super-Resolution Based Deep Learning Techniques for Panchromatic Satellite Images in Application to Pansharpening , 2020, IEEE Access.

[50]  Jocelyn Chanussot,et al.  A New Pansharpening Method Based on Spatial and Spectral Sparsity Priors , 2014, IEEE Transactions on Image Processing.

[51]  Dafang Zhuang,et al.  Survey of Multispectral Image Fusion Techniques in Remote Sensing Applications , 2011 .

[52]  Haitao Yin,et al.  Sparse representation based pansharpening with details injection model , 2015, Signal Process..

[53]  Yun Zhang,et al.  An IHS and wavelet integrated approach to improve pan-sharpening visual quality of natural colour IKONOS and QuickBird images , 2005, Inf. Fusion.

[54]  Haowen Yan,et al.  An Improved Pulse-Coupled Neural Network Model for Pansharpening , 2020, Sensors.

[55]  Tao Li,et al.  A Variational Pan-Sharpening Method Based on Spatial Fractional-Order Geometry and Spectral–Spatial Low-Rank Priors , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[56]  Ting Liu,et al.  Remote sensing image fusion via wavelet transform and sparse representation , 2015 .

[57]  Qingming Zhan,et al.  An IHS-Based Pan-Sharpening Method for Spectral Fidelity Improvement Using Ripplet Transform and Compressed Sensing , 2018, Sensors.

[58]  Manfred Ehlers,et al.  Performance of evaluation methods in image fusion , 2009, 2009 12th International Conference on Information Fusion.

[59]  Liangpei Zhang,et al.  Review of the pansharpening methods for remote sensing images based on the idea of meta-analysis: Practical discussion and challenges , 2019, Inf. Fusion.

[60]  Yun Zhang,et al.  METHODS FOR IMAGE FUSION QUALITY ASSESSMENT - A REVIEW, COMPARISON AND ANALYSIS , 2008 .

[61]  R.S. Blum,et al.  Experimental tests of image fusion for night vision , 2005, 2005 7th International Conference on Information Fusion.

[62]  A. Stein,et al.  Region- and pixel-based image fusion for disaggregation of actual evapotranspiration , 2015 .

[63]  R. J. Bhiwani,et al.  Image Fusion in Remote Sensing Applications: A Review , 2015 .

[64]  Hui Li,et al.  Assessment of Pansharpening Methods Applied to WorldView-2 Imagery Fusion , 2017, Sensors.

[65]  Junfeng Yang,et al.  PanNet: A Deep Network Architecture for Pan-Sharpening , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[66]  Jocelyn Chanussot,et al.  Context-Adaptive Pansharpening Based on Image Segmentation , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[67]  Genshe Chen,et al.  Image quality assessment for performance evaluation of image fusion , 2008, 2008 11th International Conference on Information Fusion.

[68]  Hassan Ghassemian,et al.  A Compressed-Sensing-Based Pan-Sharpening Method for Spectral Distortion Reduction , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[69]  Josaphat Tetuko Sri Sumantyo,et al.  Assessment of pan-sharpening methods applied to image fusion of remotely sensed multi-band data , 2012, Int. J. Appl. Earth Obs. Geoinformation.

[70]  Te-Ming Tu,et al.  A fast intensity-hue-saturation fusion technique with spectral adjustment for IKONOS imagery , 2004, IEEE Geoscience and Remote Sensing Letters.

[71]  Farhad Samadzadegan,et al.  Spectral and Spatial Quality assessment of IHS and Wavelet Based Pan-sharpening Techniques for High Resolution Satellite Imagery , 2018 .

[72]  S. Baronti,et al.  Remote Sensing Image Fusion , 2015 .

[73]  Lucien Wald,et al.  Analysis of Changes in Quality Assessment with Scale , 2006, 2006 9th International Conference on Information Fusion.

[74]  Jun Li,et al.  Pansharpening via Detail Injection Based Convolutional Neural Networks , 2018, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[76]  S. Baronti,et al.  Multispectral and panchromatic data fusion assessment without reference , 2008 .

[77]  Jocelyn Chanussot,et al.  A Pansharpening Method Based on the Sparse Representation of Injected Details , 2015, IEEE Geoscience and Remote Sensing Letters.

[78]  Shuyuan Yang,et al.  Pan-sharpening via deep metric learning , 2018 .

[79]  Xinghao Ding,et al.  A Variational Pan-Sharpening With Local Gradient Constraints , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[80]  Wei Wu,et al.  Pansharpening multispectral remote‐sensing images with guided filter for monitoring impact of human behavior on environment , 2018, Concurr. Comput. Pract. Exp..

[81]  Xavier Otazu,et al.  Comparison between Mallat's and the ‘à trous’ discrete wavelet transform based algorithms for the fusion of multispectral and panchromatic images , 2005 .

[82]  Liangpei Zhang,et al.  Spatial–Spectral Fusion by Combining Deep Learning and Variational Model , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[83]  Xiangyu Liu,et al.  Psgan: A Generative Adversarial Network for Remote Sensing Image Pan-Sharpening , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[84]  Giuseppe Scarpa,et al.  A Detail-Preserving Cross-Scale Learning Strategy for CNN-Based Pansharpening , 2020, Remote. Sens..

[85]  Wangmeng Zuo,et al.  Learning Deep CNN Denoiser Prior for Image Restoration , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[86]  Jocelyn Chanussot,et al.  Comparison of Pansharpening Algorithms: Outcome of the 2006 GRS-S Data-Fusion Contest , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[87]  Pragati Upadhyay,et al.  PIXEL-LEVEL IMAGE FUSION USINGBROVEY TRANSFORME AND WAVELETTRANSFORM , 2013 .

[88]  Farhad Samadzadegan,et al.  Spatial Quality Assessment of Pan-Sharpened High Resolution Satellite Imagery Based on an Automatically Estimated Edge Based Metric , 2013, Remote. Sens..

[89]  Zongxu Pan,et al.  Super-Resolution Based on Compressive Sensing and Structural Self-Similarity for Remote Sensing Images , 2013, IEEE Transactions on Geoscience and Remote Sensing.

[90]  Mehran Yazdi,et al.  Improved IHS Pan-Sharpening Method Based on Adaptive Injection of À trous Wavelet Decomposition , 2016 .

[91]  Veronika Kopacková,et al.  Testing a Modified PCA-Based Sharpening Approach for Image Fusion , 2016, Remote. Sens..

[92]  S. Sides,et al.  Comparison of three different methods to merge multiresolution and multispectral data: Landsat TM and SPOT panchromatic , 1991 .

[93]  Yonghyun Kim,et al.  Improved Additive-Wavelet Image Fusion , 2011, IEEE Geoscience and Remote Sensing Letters.

[94]  Farhad Samadzadegan,et al.  Object-based spectral quality assessment of high-resolution pan-sharpened satellite imageries: new combined fusion strategy to increase the spectral quality , 2020, Arabian Journal of Geosciences.

[95]  J. G. Liu,et al.  Smoothing Filter-based Intensity Modulation: A spectral preserve image fusion technique for improving spatial details , 2000 .

[96]  Fang Li,et al.  High-Quality Bayesian Pansharpening , 2019, IEEE Transactions on Image Processing.

[97]  Davide Cozzolino,et al.  Target-Adaptive CNN-Based Pansharpening , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[98]  A. Hegde,et al.  Evaluation of pan-sharpening methods for spatial and spectral quality , 2017 .

[99]  Xavier Otazu,et al.  Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods , 2005, IEEE Transactions on Geoscience and Remote Sensing.

[100]  Multiscale contrast image fusion scheme with performance measures , 2004 .

[101]  Francisco Eugenio,et al.  Object-based quality evaluation procedure for fused remote sensing imagery , 2017, Neurocomputing.

[102]  Manfred Ehlers,et al.  Multi-sensor image fusion for pansharpening in remote sensing , 2010 .

[103]  Naoto Yokoya,et al.  Hyperspectral Pansharpening: A Review , 2015, IEEE Geoscience and Remote Sensing Magazine.

[104]  Johannes R. Sveinsson,et al.  A New Pansharpening Algorithm Based on Total Variation , 2014, IEEE Geoscience and Remote Sensing Letters.

[105]  Mahdi Hasanlou,et al.  Quality assessment of pan-sharpening methods in high-resolution satellite images using radiometric and geometric index , 2015, Arabian Journal of Geosciences.

[106]  Wen Dou,et al.  Image Degradation for Quality Assessment of Pan-Sharpening Methods , 2018, Remote. Sens..

[107]  Yang Sun,et al.  Pan-sharpening based on common saliency feature analysis and multiscale spatial information extraction for multiple remote sensing images , 2020 .

[108]  Walaa Hassan Elashmawi,et al.  A Robust Pan-Sharpening Scheme for Improving Resolution of Satellite Images in the Domain of the Nonsubsampled Shearlet Transform , 2019 .

[109]  Davide Cozzolino,et al.  Pansharpening by Convolutional Neural Networks , 2016, Remote. Sens..

[110]  F. Samadzadegan,et al.  An object-level strategy for pan-sharpening quality assessment of high-resolution satellite imagery , 2014 .

[111]  Le Wang,et al.  ISVR: an improved synthetic variable ratio method for image fusion , 2008 .

[112]  Te-Ming Tu,et al.  A new look at IHS-like image fusion methods , 2001, Inf. Fusion.

[113]  R. Riyahi,et al.  COMPARISON OF DIFFERENT IMAGE FUSION TECHNIQUES FOR INDIVIDUAL TREE CROWN IDENTIFICATION USING QUICKBIRD IMAGES , 2009 .

[114]  Jocelyn Chanussot,et al.  A Two-Stream Multiscale Deep Learning Architecture for Pan-Sharpening , 2020, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[115]  Te-Ming Tu,et al.  Best Tradeoff for High-Resolution Image Fusion to Preserve Spatial Details and Minimize Color Distortion , 2007, IEEE Geoscience and Remote Sensing Letters.

[116]  Gwendoline Blanchet,et al.  A Survey of Pansharpening Methods with A New Band-Decoupled Variational Model , 2016, ArXiv.

[117]  Rafael García,et al.  Fusion of multispectral and panchromatic images using improved IHS and PCA mergers based on wavelet decomposition , 2004, IEEE Transactions on Geoscience and Remote Sensing.

[118]  Gaurav Sharma,et al.  A Regularized Model-Based Optimization Framework for Pan-Sharpening , 2014, IEEE Transactions on Image Processing.

[119]  Jon Atli Benediktsson,et al.  Pansharpening With Matting Model , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[120]  Christine Pohl,et al.  Multisensor image fusion in remote sensing: concepts, methods and applications , 1998 .

[121]  Munmun Baisantry,et al.  Object-based image fusion for minimization of color distortion in high-resolution satellite imagery , 2011, 2011 International Conference on Image Information Processing.

[122]  T. Ranchin,et al.  Liu 'Smoothing filter-based intensity modulation: A spectral preserve image fusion technique for improving spatial details' , 2002 .

[123]  Myungjin Choi,et al.  A new intensity-hue-saturation fusion approach to image fusion with a tradeoff parameter , 2006, IEEE Trans. Geosci. Remote. Sens..

[124]  Nicolas H. Younan,et al.  Quantitative analysis of pansharpened images , 2006 .

[125]  Thierry Ranchin,et al.  Liu 'Smoothing filter-based intensity modulation: A spectral preserve image fusion technique for improving spatial details' , 2002 .

[126]  Yun Zhang,et al.  Wavelet based image fusion techniques — An introduction, review and comparison , 2007 .

[127]  Luciano Alparone,et al.  Fast Reproducible Pansharpening Based on Instrument and Acquisition Modeling: AWLP Revisited , 2019, Remote. Sens..

[128]  Qiuming Cheng,et al.  An image fusion method based on object-oriented classification , 2012 .