Scale-Invariant Structure Saliency Selection for Fast Image Fusion

Abstract In this paper, we present a fast yet effective method for pixel-level scale-invariant image fusion in spatial domain based on the scale space theory. Specifically, we propose a scale-invariant structure saliency selection scheme based on the difference-of-Gaussian (DoG) scale space pyramid of images to build the weights or activity map. Due to the scale-invariant structure saliency selection, our method can keep both details of small size objects and the integrity information of large size objects in images. In addition, our method is very efficient since there are no complex operations involved and easy to be implemented. Experimental results demonstrate that compared to state-of-the-art image fusion methods, the proposed method yields competitive or even better results in terms of both visual quality and objective metrics, but it is much faster and therefore is appropriate to high-resolution and high-throughput image fusion tasks. Code is available at https://github.com/yiqingmy/Fusion .

[1]  Rabab Kreidieh Ward,et al.  Image Fusion With Convolutional Sparse Representation , 2016, IEEE Signal Processing Letters.

[2]  Jinde Cao,et al.  FuseGAN: Learning to Fuse Multi-Focus Image via Conditional Generative Adversarial Network , 2019, IEEE Transactions on Multimedia.

[3]  Yu Liu,et al.  Multi-focus image fusion with a deep convolutional neural network , 2017, Inf. Fusion.

[4]  Yibo Chen,et al.  Robust Multi-Focus Image Fusion Using Edge Model and Multi-Matting. , 2018, IEEE transactions on image processing : a publication of the IEEE Signal Processing Society.

[5]  Vladimir Petrovic,et al.  Objective image fusion performance measure , 2000 .

[6]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[7]  Sun Li,et al.  Multi-scale weighted gradient-based fusion for multi-focus images , 2014, Inf. Fusion.

[8]  Hadi Seyedarabi,et al.  A non-reference image fusion metric based on mutual information of image features , 2011, Comput. Electr. Eng..

[9]  Vladimir S. Petrovic,et al.  Gradient-based multiresolution image fusion , 2004, IEEE Transactions on Image Processing.

[10]  Shutao Li,et al.  Pixel-level image fusion: A survey of the state of the art , 2017, Inf. Fusion.

[11]  Zheng Liu,et al.  Statistical comparison of image fusion algorithms: Recommendations , 2017, Inf. Fusion.

[12]  Belur V. Dasarathy,et al.  Medical Image Fusion: A survey of the state of the art , 2013, Inf. Fusion.

[13]  Hui Li,et al.  DenseFuse: A Fusion Approach to Infrared and Visible Images , 2018, IEEE Transactions on Image Processing.

[14]  Qiang Zhang,et al.  Multifocus image fusion using the nonsubsampled contourlet transform , 2009, Signal Process..

[15]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[16]  Cordelia Schmid,et al.  Scale & Affine Invariant Interest Point Detectors , 2004, International Journal of Computer Vision.

[17]  A. Bovik,et al.  A universal image quality index , 2002, IEEE Signal Processing Letters.

[18]  Alan C. Bovik,et al.  Image information and visual quality , 2006, IEEE Trans. Image Process..

[19]  Shutao Li,et al.  Image Fusion With Guided Filtering , 2013, IEEE Transactions on Image Processing.

[20]  G. Qu,et al.  Information measure for performance of image fusion , 2002 .

[21]  Shutao Li,et al.  Multifocus Image Fusion and Restoration With Sparse Representation , 2010, IEEE Transactions on Instrumentation and Measurement.

[22]  Rick S. Blum,et al.  Robust sparse representation based multi-focus image fusion with dictionary construction and local spatial consistency , 2018, Pattern Recognit..

[23]  C. L. Philip Chen,et al.  Multifocus image fusion with enhanced linear spectral clustering and fast depth map estimation , 2018, Neurocomputing.

[24]  Michael F. Cohen,et al.  Digital photography with flash and no-flash image pairs , 2004, ACM Trans. Graph..

[25]  Yu Liu,et al.  Multi-focus image fusion with dense SIFT , 2015, Inf. Fusion.

[26]  Krystian Mikolajczyk,et al.  Detection of local features invariant to affines transformations , 2002 .

[27]  Jian Sun,et al.  Guided Image Filtering , 2010, ECCV.

[28]  T. Lindeberg,et al.  Scale-Space Theory : A Basic Tool for Analysing Structures at Different Scales , 1994 .

[29]  Shutao Li,et al.  Image matting for fusion of multi-focus images in dynamic scenes , 2013, Inf. Fusion.

[30]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[31]  Liron Pantanowitz,et al.  The impact of digital imaging in the field of cytopathology , 2009, CytoJournal.

[32]  R. Venkatesh Babu,et al.  DeepFuse: A Deep Unsupervised Approach for Exposure Fusion with Extreme Exposure Image Pairs , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[33]  Anup Basu,et al.  Cross-Scale Coefficient Selection for Volumetric Medical Image Fusion , 2013, IEEE Transactions on Biomedical Engineering.

[34]  Yu Liu,et al.  A general framework for image fusion based on multi-scale transform and sparse representation , 2015, Inf. Fusion.

[35]  D. Solomon,et al.  The Bethesda system for reporting cervical cytology : definitions, criteria, and explanatory notes , 2004 .

[36]  Xiaohua Xia,et al.  Multi-focus image fusion based on probability filtering and region correction , 2018, Signal Process..

[37]  Hassan Ghassemian,et al.  Combining the spectral PCA and spatial PCA fusion methods by an optimal filter , 2016, Inf. Fusion.

[38]  M. Hossny,et al.  Comments on 'Information measure for performance of image fusion' , 2008 .

[39]  Huiqian Du,et al.  Structure tensor and nonsubsampled shearlet transform based algorithm for CT and MRI image fusion , 2017, Neurocomputing.

[40]  Ke Lu,et al.  An overview of multi-modal medical image fusion , 2016, Neurocomputing.

[41]  Xin Liu,et al.  A novel similarity based quality metric for image fusion , 2008, Inf. Fusion.

[42]  Rabab Kreidieh Ward,et al.  Deep learning for pixel-level image fusion: Recent advances and future prospects , 2018, Inf. Fusion.

[43]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[44]  Tania Stathaki,et al.  Image Fusion: Algorithms and Applications , 2008 .

[45]  Nikos Komodakis,et al.  Learning to compare image patches via convolutional neural networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Jan Kautz,et al.  Exposure Fusion , 2009, 15th Pacific Conference on Computer Graphics and Applications (PG'07).

[47]  Ivor W. Tsang,et al.  Fusing images with different focuses using support vector machines , 2004, IEEE Transactions on Neural Networks.

[48]  Yu Zhang,et al.  Boundary finding based multi-focus image fusion through multi-scale morphological focus-measure , 2017, Inf. Fusion.

[49]  Ali Aghagolzadeh,et al.  Ensemble of CNN for multi-focus image fusion , 2019, Inf. Fusion.

[50]  Bo Wang,et al.  Multi-focus image fusion using boosted random walks-based algorithm with two-scale focus maps , 2019, Neurocomputing.

[51]  Shutao Li,et al.  Performance comparison of different multi-resolution transforms for image fusion , 2011, Inf. Fusion.

[52]  Shesheng Gao,et al.  Image Segmentation-Based Multi-Focus Image Fusion Through Multi-Scale Convolutional Neural Network , 2017, IEEE Access.

[53]  Xuejie Zhang,et al.  Multi-focus: Focused region finding and multi-scale transform for image fusion , 2018, Neurocomputing.

[54]  Ananda S. Chowdhury,et al.  Steerable local frequency based multispectral multifocus image fusion , 2015, Inf. Fusion.

[55]  Stavri G. Nikolov,et al.  Image fusion: Advances in the state of the art , 2007, Inf. Fusion.