Fusion of medical images based on salient features extraction by PSO optimized fuzzy logic in NSST domain

Abstract The aim of multimodal medical image fusion is to extract information from different modal images into a single one so that the single fusion image contains the salient features of the source images to the maximum extent. Different medical imaging modalities such as CT and MRI are presented as different visual morphology, which often show different complementary salient features in clinical diagnosis. According to this characteristic, in order to fuse different salient features of multimodal medical images into a single image and provide a new dimension of information for clinical diagnosis to improve the diagnosis accuracy, an improved image fusion algorithm based on visual salience detection is proposed in this paper. The visual salience of two registered source images is calculated by the GBVS algorithm, and the low-frequency and high-frequency sub-bands are obtained by decomposing the source images in NSST domain. For the low-frequency sub-bands, fuzzy logic system is used to obtain the respective weights of the fused low-frequency sub-band with the local energy and GBVS graph as input. For the high-frequency sub-bands, the NSML values of each sub-band are calculated and compared to obtain the fused high-frequency sub-band. The final fused image is obtained by the inverse NSST transformation. In addition, PSO algorithm is used to optimize the membership function of fuzzy logic system for better adaption to medical images and feature extraction. By applying this multimodal medical image fusion method, the visual quality of the image is effectively improved and the salient features of tissues are preserved well. Experiments on fusion of grayscale, color and ultrasound multimodal medical images show that the proposed method has advantages on retention of image salient features. Compared with other models, the fused image obtained by the proposed method has higher objective indexes with clearer edge contour, higher overall contrast, and no ringing effect and artifacts.

[1]  H. N. Suma,et al.  Fusion of CT and PET Medical Images Using Hybrid Algorithm DWT-DCT-PCA , 2015, 2015 2nd International Conference on Information Science and Security (ICISS).

[2]  G. Easley,et al.  Sparse directional image representations using the discrete shearlet transform , 2008 .

[3]  Rabab Kreidieh Ward,et al.  Deep learning for pixel-level image fusion: Recent advances and future prospects , 2018, Inf. Fusion.

[4]  Ashish Khare,et al.  Fusion of multimodal medical images using Daubechies complex wavelet transform - A multiresolution approach , 2014, Inf. Fusion.

[5]  A Novel Fusion Method for Multi-Focus Images , 2010 .

[6]  Vladimir Petrovic,et al.  Objective image fusion performance measure , 2000 .

[7]  Yang Lei,et al.  Multimodal Sensor Medical Image Fusion Based on Local Difference in Non-Subsampled Domain , 2019, IEEE Transactions on Instrumentation and Measurement.

[8]  Yide Ma,et al.  Multimodal Medical Image Fusion Framework Based on Simplified PCNN in Nonsubsampled Contourlet Transform Domain , 2013, J. Multim..

[9]  Ke Lu,et al.  An overview of multi-modal medical image fusion , 2016, Neurocomputing.

[10]  Alan C. Bovik,et al.  Image information and visual quality , 2006, IEEE Trans. Image Process..

[11]  R. S. Anand,et al.  Multimodal Medical Image Fusion Using Hybrid Layer Decomposition With CNN-Based Feature Mapping and Structural Clustering , 2020, IEEE Transactions on Instrumentation and Measurement.

[12]  Xun Chen,et al.  Medical Image Fusion With Parameter-Adaptive Pulse Coupled Neural Network in Nonsubsampled Shearlet Transform Domain , 2019, IEEE Transactions on Instrumentation and Measurement.

[13]  Yu Liu,et al.  IFCNN: A general image fusion framework based on convolutional neural network , 2020, Inf. Fusion.

[14]  Maurice Clerc,et al.  The particle swarm - explosion, stability, and convergence in a multidimensional complex space , 2002, IEEE Trans. Evol. Comput..

[15]  Jiayi Ma,et al.  Infrared and visible image fusion methods and applications: A survey , 2018, Inf. Fusion.

[16]  Durga Prasad Bavirisetti,et al.  Multi-focus image fusion using multi-scale image decomposition and saliency detection , 2016, Ain Shams Engineering Journal.

[17]  Yaqin Zhao,et al.  Multi-modality medical images fusion based on local-features fuzzy sets and novel sum-modified-Laplacian in non-subsampled shearlet transform domain , 2020, Biomed. Signal Process. Control..

[18]  Zhen Chao,et al.  Multi-modality image fusion based on enhanced fuzzy radial basis function neural networks. , 2018, Physica medica : PM : an international journal devoted to the applications of physics to medicine and biology : official journal of the Italian Association of Biomedical Physics.

[19]  Shutao Li,et al.  Multifocus Image Fusion and Restoration With Sparse Representation , 2010, IEEE Transactions on Instrumentation and Measurement.

[20]  Frans van den Bergh,et al.  An analysis of particle swarm optimizers , 2002 .

[21]  Henk J. A. M. Heijmans,et al.  A new quality metric for image fusion , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[22]  Bin Wang,et al.  Multi-band images synchronous fusion based on NSST and fuzzy logical inference , 2019, Infrared Physics & Technology.

[23]  Yu Liu,et al.  Multi-focus image fusion with a deep convolutional neural network , 2017, Inf. Fusion.

[24]  Syed Omer Gilani,et al.  The Effect of Eye Movements in Response to Different Types of Scenes Using a Graph-Based Visual Saliency Algorithm , 2019 .

[25]  Yuji Iwahori,et al.  Non-subsampled shearlet transform-based image fusion using modified weighted saliency and local difference , 2018, Multimedia Tools and Applications.

[26]  L. Yang,et al.  Multimodality medical image fusion based on multiscale geometric analysis of contourlet transform , 2008, Neurocomputing.

[27]  Yi Liu,et al.  Sparse representation based multi-sensor image fusion for multi-focus and multi-modality images: A review , 2018, Inf. Fusion.

[28]  Hamid Reza Shahdoosti,et al.  MRI and PET/SPECT image fusion at feature level using ant colony based segmentation , 2019, Biomed. Signal Process. Control..

[29]  Z. Hasan.  A Survey on Shari’Ah Governance Practices in Malaysia, GCC Countries and the UK , 2011 .

[30]  A. Bovik,et al.  A universal image quality index , 2002, IEEE Signal Processing Letters.

[31]  Minh N. Do,et al.  Ieee Transactions on Image Processing the Contourlet Transform: an Efficient Directional Multiresolution Image Representation , 2022 .

[32]  Minh N. Do,et al.  The Nonsubsampled Contourlet Transform: Theory, Design, and Applications , 2006, IEEE Transactions on Image Processing.

[33]  Yi Chai,et al.  A novel multi-modality image fusion method based on image decomposition and sparse representation , 2017, Inf. Sci..

[34]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[35]  Belur V. Dasarathy,et al.  Medical Image Fusion: A survey of the state of the art , 2013, Inf. Fusion.

[36]  Rabab K. Ward,et al.  Medical Image Fusion via Convolutional Sparsity Based Morphological Component Analysis , 2019, IEEE Signal Processing Letters.

[37]  Huiqian Du,et al.  Structure tensor and nonsubsampled shearlet transform based algorithm for CT and MRI image fusion , 2017, Neurocomputing.

[38]  Huiqian Du,et al.  Multi-modality medical image fusion based on image decomposition framework and nonsubsampled shearlet transform , 2018, Biomed. Signal Process. Control..

[39]  Donald K. Wedding.  Fuzzy sets and fuzzy logic: Theory and applications: George J. Klir and Bo Yuan, Prentice Hall, 1995, ISBN 0-13-101171-5, pp. 574 , 1997 .

[40]  Jipsha Mariam Dolly,et al.  A Survey on Different Multimodal Medical Image Fusion Techniques and Methods , 2019, 2019 1st International Conference on Innovations in Information and Communication Technology (ICIICT).

[41]  G. Qu,et al.  Information measure for performance of image fusion , 2002 .

[42]  Yu Guo,et al.  A novel image fusion algorithm based on nonsubsampled shearlet transform , 2014 .

[43]  Cheng Zhao,et al.  Infrared and visible image fusion method based on rolling guidance filter and NSST , 2019, Int. J. Wavelets Multiresolution Inf. Process..