Super Resolution Infrared Thermal Imaging Using Pansharpening Algorithms: Quantitative Assessment and Application to UAV Thermal Imaging

The lack of high-resolution thermal images is a limiting factor in the fusion with other sensors with a higher resolution. Different families of algorithms have been designed in the field of remote sensors to fuse panchromatic images with multispectral images from satellite platforms, in a process known as pansharpening. Attempts have been made to transfer these pansharpening algorithms to thermal images in the case of satellite sensors. Our work analyses the potential of these algorithms when applied to thermal images from unmanned aerial vehicles (UAVs). We present a comparison, by means of a quantitative procedure, of these pansharpening methods in satellite images when they are applied to fuse high-resolution images with thermal images obtained from UAVs, in order to be able to choose the method that offers the best quantitative results. This analysis, which allows the objective selection of which method to use with this type of images, has not been done until now. This algorithm selection is used here to fuse images from thermal sensors on UAVs with other images from different sensors for the documentation of heritage, but it has applications in many other fields.

[1]  S. Sides,et al.  Comparison of three different methods to merge multiresolution and multispectral data: Landsat TM and SPOT panchromatic , 1991 .

[2]  Lucien Wald,et al.  Quality of high resolution synthesised images: Is there a simple criterion ? , 2000 .

[3]  Luca Tavasci,et al.  A multi-image super-resolution algorithm applied to thermal imagery , 2019, Applied Geomatics.

[4]  Te-Ming Tu,et al.  A new look at IHS-like image fusion methods , 2001, Inf. Fusion.

[5]  Yu Lei,et al.  An Improved ORB Algorithm of Extracting and Matching Features , 2015 .

[6]  Naoto Yokoya,et al.  Hyperspectral and Multispectral Data Fusion: A comparative review of the recent literature , 2017, IEEE Geoscience and Remote Sensing Magazine.

[7]  Chulhee Lee,et al.  Fast and Efficient Panchromatic Sharpening , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[8]  Ni-Bin Chang,et al.  Multisensor Data Fusion and Machine Learning for Environmental Remote Sensing , 2018 .

[9]  Russell C. Hardie,et al.  Hyperspectral resolution enhancement using high-resolution multispectral imagery with arbitrary response functions , 2005, IEEE Transactions on Geoscience and Remote Sensing.

[10]  Ji Zhou,et al.  Disaggregation of remotely sensed land surface temperature: Literature survey, taxonomy, issues, and caveats , 2013 .

[11]  L. Alparone,et al.  An MTF-based spectral distortion minimizing model for pan-sharpening of very high resolution multispectral images of urban areas , 2003, 2003 2nd GRSS/ISPRS Joint Workshop on Remote Sensing and Data Fusion over Urban Areas.

[12]  Nicolas H. Younan,et al.  Quality analysis of pansharpened images , 2004, IGARSS 2004. 2004 IEEE International Geoscience and Remote Sensing Symposium.

[13]  Sharon A. Robinson,et al.  Spatial Co-Registration of Ultra-High Resolution Visible, Multispectral and Thermal Images Acquired with a Micro-UAV over Antarctic Moss Beds , 2014, Remote. Sens..

[14]  L. Wald,et al.  Fusion of high spatial and spectral resolution images : The ARSIS concept and its implementation , 2000 .

[15]  Costin-Anton Boiangiu,et al.  A Study of Image Upsampling and Downsampling Filters , 2019, Comput..

[16]  Naoto Yokoya,et al.  Hyperspectral Pansharpening: A Review , 2015, IEEE Geoscience and Remote Sensing Magazine.

[17]  Andrea Garzelli,et al.  Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis , 2002, IEEE Trans. Geosci. Remote. Sens..

[18]  Hazem M. El-Bakry,et al.  Image Stitching System Based on ORB Feature-Based Technique and Compensation Blending , 2015 .

[19]  Aleksandra Pizurica,et al.  Two-stage fusion of thermal hyperspectral and visible RGB image by PCA and guided filter , 2015, 2015 7th Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS).

[20]  Johannes R. Sveinsson,et al.  Sentinel-2 Image Fusion Using a Deep Residual Network , 2018, Remote. Sens..

[21]  Jocelyn Chanussot,et al.  A Critical Comparison Among Pansharpening Algorithms , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[22]  Bruno Aiazzi,et al.  A Comparison Between Global and Context-Adaptive Pansharpening of Multispectral Images , 2009, IEEE Geoscience and Remote Sensing Letters.

[23]  Xiaohui Chen,et al.  Color guided thermal image super resolution , 2016, 2016 Visual Communications and Image Processing (VCIP).

[24]  Olivier Debeir,et al.  Multimodal Sensor Fusion In Single Thermal image Super-Resolution , 2018, ACCV Workshops.

[25]  Chiman Kwan,et al.  Blind Quality Assessment of Fused WorldView-3 Images by Using the Combinations of Pansharpening and Hypersharpening Paradigms , 2017, IEEE Geoscience and Remote Sensing Letters.

[26]  Neal R. Butler,et al.  Performance limits of uncooled VOx microbolometer focal plane arrays , 2004, SPIE Defense + Commercial Sensing.

[27]  S. Baronti,et al.  Remote Sensing Image Fusion , 2015 .

[28]  Bruno Aiazzi,et al.  Hyper-Sharpening: A First Approach on SIM-GA Data , 2015, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[29]  Hyung-Sup Jung,et al.  Multi-Sensor Fusion of Landsat 8 Thermal Infrared (TIR) and Panchromatic (PAN) Images , 2014, Sensors.

[30]  J. G. Liu,et al.  Smoothing Filter-based Intensity Modulation : a spectral preserve image fusion technique for improving spatial details , 2001 .

[31]  Naoto Yokoya,et al.  Texture-Guided Multisensor Superresolution for Remotely Sensed Images , 2017, Remote. Sens..

[32]  Luciano Alparone,et al.  MTF-tailored Multiscale Fusion of High-resolution MS and Pan Imagery , 2006 .

[33]  L. Wald,et al.  Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images , 1997 .

[34]  Andrea Garzelli,et al.  Optimal MMSE Pan Sharpening of Very High Resolution Multispectral Images , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[35]  Bayarri,et al.  Hyperspectral Imaging Techniques for the Study, Conservation and Management of Rock Art , 2019 .

[36]  Jie Li,et al.  Image super-resolution: The techniques, applications, and future , 2016, Signal Process..

[37]  Jocelyn Chanussot,et al.  Indusion: Fusion of Multispectral and Panchromatic Images Using the Induction Scaling Technique , 2008, IEEE Geoscience and Remote Sensing Letters.

[38]  A. Bovik,et al.  A universal image quality index , 2002, IEEE Signal Processing Letters.

[39]  Kiyun Yu,et al.  A New Adaptive Component-Substitution-Based Satellite Image Fusion by Using Partial Replacement , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[40]  T. Ranchin,et al.  Liu 'Smoothing filter-based intensity modulation: A spectral preserve image fusion technique for improving spatial details' , 2002 .

[41]  Pedro Arias,et al.  Automation of thermographic 3D modelling through image fusion and image matching techniques , 2011 .

[42]  Pedro Arias,et al.  Calibration and verification of thermographic cameras for geometric measurements , 2011 .

[43]  Ruiliang Pu,et al.  Assessing scaling effect in downscaling land surface temperature in a heterogenous urban environment , 2021, Int. J. Appl. Earth Obs. Geoinformation.

[44]  Samuel Ortega-Farías,et al.  Automatic Coregistration Algorithm to Remove Canopy Shaded Pixels in UAV-Borne Thermal Images to Improve the Estimation of Crop Water Stress Index of a Drip-Irrigated Cabernet Sauvignon Vineyard , 2018, Sensors.