Multilevel Features Convolutional Neural Network for Multifocus Image Fusion

Multifocus image fusion is an important technique that aims to generate a single clean image by fusing multiple input images. In this paper, we propose a novel multilevel features convolutional neural network (MLFCNN) architecture for image fusion. In the MLFCNN model, all features learned from previous layers are passed to the subsequent layer. Inside every path between the previous layer and the subsequent layer, we add a 1 × 1 convolution module to reduce the redundancy. In our method, the source images first are fed to our pre-trained MLFCNN model to obtain the initial focus map. Then, the initial focus map is performed by morphological opening and closing operations and followed by a Gaussian filter to obtain the final decision map. Finally, the fused all-in-focus image is generated based on a weighted-sum strategy with the decision map. The experimental results demonstrate that the proposed method outperforms some state-of-the-art image fusion algorithms in terms of both qualitative and objective evaluations.

[1]  B. S. Manjunath,et al.  Multisensor Image Fusion Using the Wavelet Transform , 1995, CVGIP Graph. Model. Image Process..

[2]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[3]  Nannan Yu,et al.  Image Features Extraction and Fusion Based on Joint Sparse Representation , 2011, IEEE Journal of Selected Topics in Signal Processing.

[4]  Shutao Li,et al.  Image matting for fusion of multi-focus images in dynamic scenes , 2013, Inf. Fusion.

[5]  B. S. Manjunath,et al.  Multi-sensor image fusion using the wavelet transform , 1994, Proceedings of 1st International Conference on Image Processing.

[6]  Shesheng Gao,et al.  Image Segmentation-Based Multi-Focus Image Fusion Through Multi-Scale Convolutional Neural Network , 2017, IEEE Access.

[7]  Xin Liu,et al.  A novel similarity based quality metric for image fusion , 2008, Inf. Fusion.

[8]  Shutao Li,et al.  Image Fusion With Guided Filtering , 2013, IEEE Transactions on Image Processing.

[9]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Shutao Li,et al.  Pixel-level image fusion with simultaneous orthogonal matching pursuit , 2012, Inf. Fusion.

[11]  Jing Tian,et al.  Adaptive multi-focus image fusion using a wavelet-based statistical sharpness measure , 2012, Signal Process..

[12]  G. Qu,et al.  Information measure for performance of image fusion , 2002 .

[13]  Edward H. Adelson,et al.  The Laplacian Pyramid as a Compact Image Code , 1983, IEEE Trans. Commun..

[14]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[15]  Qiang Chen,et al.  Network In Network , 2013, ICLR.

[16]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[17]  Yu Zhang,et al.  Boundary finding based multi-focus image fusion through multi-scale morphological focus-measure , 2017, Inf. Fusion.

[18]  Yu Liu,et al.  Multi-focus image fusion with a deep convolutional neural network , 2017, Inf. Fusion.

[19]  Pan Lin,et al.  Multimodal Sensor Medical Image Fusion Based on Type-2 Fuzzy Logic in NSCT Domain , 2016, IEEE Sensors Journal.

[20]  Luciano Alparone,et al.  Remote sensing image fusion using the curvelet transform , 2007, Inf. Fusion.

[21]  Yu Liu,et al.  Simultaneous image fusion and denoising with adaptive sparse representation , 2015, IET Image Process..

[22]  Yuan Yan Tang,et al.  Multi-focus image fusion based on the neighbor distance , 2013, Pattern Recognit..

[23]  Pan Lin,et al.  Multifocus Image Fusion Based on NSCT and Focused Area Detection , 2014, IEEE Sensors Journal.

[24]  Shadrokh Samavi,et al.  Multi-focus image fusion using dictionary-based sparse representation , 2015, Inf. Fusion.

[25]  Shutao Li,et al.  Pixel-level image fusion: A survey of the state of the art , 2017, Inf. Fusion.

[26]  Sun Li,et al.  Multi-scale weighted gradient-based fusion for multi-focus images , 2014, Inf. Fusion.

[27]  Yu Liu,et al.  A general framework for image fusion based on multi-scale transform and sparse representation , 2015, Inf. Fusion.

[28]  B. K. Shreyamsha Kumar,et al.  Image fusion based on pixel significance using cross bilateral filter , 2013, Signal, Image and Video Processing.

[29]  Veysel Aslantas,et al.  Fusion of multi-focus images using differential evolution algorithm , 2010, Expert Syst. Appl..

[30]  Pan Lin,et al.  Multiple Visual Features Measurement With Gradient Domain Guided Filtering for Multisensor Image Fusion , 2017, IEEE Transactions on Instrumentation and Measurement.

[31]  Yu Liu,et al.  Multi-focus image fusion with dense SIFT , 2015, Inf. Fusion.

[32]  Yu Zhang,et al.  Quadtree-based multi-focus image fusion using a weighted focus-measure , 2015, Inf. Fusion.

[33]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[34]  Vladimir Petrovic,et al.  Objective image fusion performance measure , 2000 .

[35]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[36]  Vladimir S. Petrovic,et al.  Gradient-based multiresolution image fusion , 2004, IEEE Transactions on Image Processing.

[37]  Shutao Li,et al.  Multifocus Image Fusion and Restoration With Sparse Representation , 2010, IEEE Transactions on Instrumentation and Measurement.

[38]  Yibo Chen,et al.  Robust Multi-Focus Image Fusion Using Edge Model and Multi-Matting. , 2018, IEEE transactions on image processing : a publication of the IEEE Signal Processing Society.