Fusion of Thermal Infrared and Visible Images Based on Multi-scale Transform and Sparse Representation

Due to the differences between the visible and thermal infrared images, the combination of these two types of images leads to better understanding of the characteristics of targets and the environment. Thermal infrared images are really in distinguishing targets from the background based on the radiation differences and land surface temperature (LST) calculation. However, their spatial resolution is relatively low, making it challenging to detect targets. Image fusion is an efficient method to be employed to enhance spatial resolution of the thermal bands through fusing these images with high spatial resolution visible images. Multi-scale transforms (MST) and sparse representation (SR) are widely used in image fusion. To improve the performance of image fusion, these two types of methods can be combined. In this paper, nine image fusion methods based on the multi-scale transform and sparse representation, namely Laplacian pyramid (LP), ratio of low-pass pyramid (RP), wavelet transform (Wavelet), dual-tree complex wavelet transform (DTCWT), curvelet transform (CVT), nonsubsampled contourlet transform (NSCT), sparse representation (SR), hybrid sparse representation and Laplacian pyramid methods (LP-SR) and hybrid sparse representation and NSCT methods (NSCT-SR) are tested on FLIR and landsat-8 thermal infrared and visible images. To evaluate the performance of different image fusion methods we used following three the quantitative evaluation metrics: entropy (EN), mutual information (MI), and gradient based fusion metric Q . Despite the lack of spectral coverage between the visible and thermal infrared bands of Landsat 8, quantitative evaluation metrics showed that the hybrid LP-SR method provides the best result (EN=7.362, MI=2.605, Q =0.531) and fused images have a better visual quality. This method improves spatial details along with preserving the thermal radiation information. This method is followed by RP, LP, and NSCT methods. Similar results were achieved in FLIR images.

[1]  Yi Liu,et al.  Sparse representation based multi-sensor image fusion for multi-focus and multi-modality images: A review , 2018, Inf. Fusion.

[2]  Xiaoqing Luo,et al.  Image Fusion Using Quaternion Wavelet Transform and Multiple Features , 2017, IEEE Access.

[3]  M. Elad,et al.  $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation , 2006, IEEE Transactions on Signal Processing.

[4]  Shutao Li,et al.  Performance comparison of different multi-resolution transforms for image fusion , 2011, Inf. Fusion.

[5]  Nong Sang,et al.  Attention-based hierarchical fusion of visible and infrared images , 2015 .

[6]  Jun Huang,et al.  Infrared and visible image fusion via saliency analysis and local edge-preserving multi-scale decomposition. , 2017, Journal of the Optical Society of America. A, Optics, image science, and vision.

[7]  A Rajeshwari,et al.  ESTIMATION OF LAND SURFACE TEMPERATURE OF DINDIGUL DISTRICT USING LANDSAT 8 DATA , 2014 .

[8]  Kjersti Engan,et al.  Multi-frame compression: theory and design , 2000, Signal Process..

[9]  Yu Liu,et al.  A general framework for image fusion based on multi-scale transform and sparse representation , 2015, Inf. Fusion.

[10]  Jing Yang,et al.  Image Fusion for Spatial Enhancement of Hyperspectral Image via Pixel Group Based Non-Local Sparse Representation , 2017, Remote. Sens..

[11]  Ting Liu,et al.  Remote sensing image fusion via wavelet transform and sparse representation , 2015 .

[12]  B. S. Manjunath,et al.  Multisensor Image Fusion Using the Wavelet Transform , 1995, CVGIP Graph. Model. Image Process..

[13]  Yi Chai,et al.  A Novel Geometric Dictionary Construction Approach for Sparse Representation Based Image Fusion , 2017, Entropy.

[14]  Bruno Aiazzi,et al.  SPATIAL ENHANCEMENT OF TIR ASTER DATA VIA VNIR IMAGES AND GENERALIZED LAPLACIAN DECOMPOSITION , 2005 .

[15]  Victor Alchanatis,et al.  Image fusion of visible and thermal images for fruit detection. , 2009 .

[16]  Fang Liu,et al.  Fusion of infrared and visual images based on contrast pyramid directional filter banks using clonal selection optimizing , 2008 .

[17]  Shutao Li,et al.  Remote Sensing Image Fusion via Sparse Representations Over Learned Dictionaries , 2013, IEEE Transactions on Geoscience and Remote Sensing.

[18]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[19]  Jiayi Ma,et al.  Infrared and visible image fusion methods and applications: A survey , 2018, Inf. Fusion.

[20]  Yue Qi,et al.  Infrared and visible image fusion method based on saliency detection in sparse domain , 2017 .

[21]  Yide Ma,et al.  Image fusion based on steerable pyramid and PCNN , 2009, 2009 Second International Conference on the Applications of Digital Information and Web Technologies.