Image Fusion Based on Machine Learning and Deep Learning

Machine learning and deep learning are finding applications in various computer vision problems such as object recognition, detection, and visual tracking. In addition, in computer vision, it is quite common to fuse information acquired in different spectral ranges, focusing, and lighting conditions to know more details of a particular scene. Hence, image fusion using machine learning especially deep learning would be a hot research topic in upcoming years due to recent advancements in both software and computing capabilities. In this chapter, we made an effort to present machine learning- and deep learning-based fusion in a simple manner for fundamental understanding. In Sect. 7.1, a general introduction to AI and its general classification is presented. Section 7.2 gives an overview of machine learning from basic definitions to advanced concepts. Image fusion based on machine learning is explained in Sect. 7.3. In Sect. 7.4, important and useful concepts of deep learning are described. Section 7.5 gives an overview of state-of-the-art deep learning based image fusion. Section 7.6 presents the future scope. Finally, Sect. 7.7 concludes the chapter.

[1]  Avrim Blum,et al.  The Bottleneck , 2021, Monopsony Capitalism.

[2]  Shuyuan Yang,et al.  Contourlet hidden Markov Tree and clarity-saliency driven PCNN based remote sensing images fusion , 2012, Appl. Soft Comput..

[3]  Geoffrey E. Hinton,et al.  Restricted Boltzmann machines for collaborative filtering , 2007, ICML '07.

[4]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[5]  S. Süsstrunk,et al.  Frequency-tuned salient region detection , 2009, CVPR 2009.

[6]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Jürgen Schmidhuber,et al.  Multi-column deep neural network for traffic sign classification , 2012, Neural Networks.

[8]  N. Ahmed,et al.  Discrete Cosine Transform , 1996 .

[9]  Ming Yang,et al.  3D Convolutional Neural Networks for Human Action Recognition , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Charu C. Aggarwal,et al.  Data Mining: The Textbook , 2015 .

[11]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[12]  Li Jie,et al.  A Novel Support Vector Machine-Based Multifocus Image Fusion Algorithm , 2006, 2006 International Conference on Communications, Circuits and Systems.

[13]  Yang Lei,et al.  Visible and infrared image fusion using NSST and deep Boltzmann machine , 2018 .

[14]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[15]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[16]  Jing Tian,et al.  Improving Empirical Mode Decomposition Using Support Vector Machines for Multifocus Image Fusion , 2008, Sensors.

[17]  Jeonghee Yi,et al.  Sentiment analysis: capturing favorability using natural language processing , 2003, K-CAP '03.

[18]  Xiang Zhang,et al.  OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks , 2013, ICLR.

[19]  Yaonan Wang,et al.  Combination of images with diverse focuses using the spatial frequency , 2001, Inf. Fusion.

[20]  Oliver Rockinger,et al.  Image sequence fusion using a shift-invariant wavelet transform , 1997, Proceedings of International Conference on Image Processing.

[21]  Ah Chung Tsoi,et al.  Face recognition: a convolutional neural-network approach , 1997, IEEE Trans. Neural Networks.

[22]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[23]  Yorick Wilks,et al.  Named Entity Recognition from Diverse Text Types , 2001 .

[24]  László Lovász,et al.  Random Walks on Graphs: A Survey , 1993 .

[25]  Rabab Kreidieh Ward,et al.  Deep learning for pixel-level image fusion: Recent advances and future prospects , 2018, Inf. Fusion.

[26]  Yaonan Wang,et al.  Multifocus image fusion using artificial neural networks , 2002, Pattern Recognit. Lett..

[27]  Zhongliang Jing,et al.  Evaluation of focus measures in multi-focus image fusion , 2007, Pattern Recognit. Lett..

[28]  Yu Liu,et al.  A medical image fusion method based on convolutional neural networks , 2017, 2017 20th International Conference on Information Fusion (Fusion).

[29]  Ivor W. Tsang,et al.  Fusing images with different focuses using support vector machines , 2004, IEEE Transactions on Neural Networks.

[30]  Hong Jiang,et al.  Fuzzy image fusion based on modified Self-Generating Neural Network , 2011, Expert Syst. Appl..

[31]  Christine Pohl,et al.  Multisensor image fusion in remote sensing: concepts, methods and applications , 1998 .

[32]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[33]  Ling Shao,et al.  Recent advances and trends in visual tracking: A review , 2011, Neurocomputing.

[34]  Alexander Toet,et al.  Multiscale image fusion through guided filtering , 2016, Security + Defence.

[35]  Wei Cai,et al.  A region-based multi-sensor image fusion scheme using pulse-coupled neural network , 2006, Pattern Recognit. Lett..

[36]  Jitendra Malik,et al.  Scale-Space and Edge Detection Using Anisotropic Diffusion , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[37]  Pascal Vincent,et al.  Contractive Auto-Encoders: Explicit Invariance During Feature Extraction , 2011, ICML.

[38]  B. K. Shreyamsha Kumar,et al.  Image fusion based on pixel significance using cross bilateral filter , 2013, Signal, Image and Video Processing.

[39]  Jason Jianjun Gu,et al.  Multi-focus image fusion using PCNN , 2010, Pattern Recognit..

[40]  Arthur L. Samuel,et al.  Programming Computers to Play Games , 1960, Adv. Comput..

[41]  Durga Prasad Bavirisetti,et al.  Fusion of Infrared and Visible Sensor Images Based on Anisotropic Diffusion and Karhunen-Loeve Transform , 2016, IEEE Sensors Journal.

[42]  Yu Liu,et al.  Multi-focus image fusion with a deep convolutional neural network , 2017, Inf. Fusion.

[43]  Andrew G. Howard,et al.  Some Improvements on Deep Convolutional Neural Network Based Image Classification , 2013, ICLR.

[44]  Ravindra Dhuli,et al.  Two-scale image fusion of visible and infrared images using saliency detection , 2016 .

[45]  Nikolaos Mitianoudis,et al.  Pixel-based and region-based image fusion schemes using ICA bases , 2007, Inf. Fusion.

[46]  Arun Ross,et al.  An introduction to biometric recognition , 2004, IEEE Transactions on Circuits and Systems for Video Technology.

[47]  Christopher Potts,et al.  Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank , 2013, EMNLP.

[48]  Hassan Ghassemian,et al.  A new pansharpening method using multi resolution analysis framework and deep neural networks , 2017, 2017 3rd International Conference on Pattern Recognition and Image Analysis (IPRIA).

[49]  Ravi Ramamoorthi,et al.  Deep high dynamic range imaging of dynamic scenes , 2017, ACM Trans. Graph..

[50]  Joan Bruna,et al.  Intriguing properties of neural networks , 2013, ICLR.

[51]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[52]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[53]  Alexander Toet,et al.  Hierarchical image fusion , 1990, Machine Vision and Applications.

[54]  B. Jack Copeland,et al.  Artificial Intelligence: A Philosophical Introduction , 1993 .

[55]  Shichao Zhang,et al.  Efficient kNN classification algorithm for big data , 2016, Neurocomputing.

[56]  Erik Lindholm,et al.  NVIDIA Tesla: A Unified Graphics and Computing Architecture , 2008, IEEE Micro.

[57]  Jia Yonghong,et al.  Fusion of Landsat TM and SAR Images Based on Principal Component Analysis , 2012 .

[58]  Alexander Toet,et al.  Image fusion by a ration of low-pass pyramid , 1989, Pattern Recognit. Lett..

[59]  Daniel Marcu.  The rhetorical parsing of natural language texts , 1997 .

[60]  Paul S. Fisher,et al.  Image quality measures and their performance , 1995, IEEE Trans. Commun..

[61]  Sarat Kumar Sahoo,et al.  Pulse coupled neural networks and its applications , 2014, Expert Syst. Appl..

[62]  B. S. Manjunath,et al.  Multisensor Image Fusion Using the Wavelet Transform , 1995, CVGIP Graph. Model. Image Process..

[63]  C. Burrus,et al.  Introduction to Wavelets and Wavelet Transforms: A Primer , 1997 .

[64]  Alexander Toet,et al.  Merging thermal and visual images by a contrast pyramid , 1989 .

[65]  Kai-Lung Hua,et al.  A novel multi-focus image fusion algorithm based on random walks , 2014, J. Vis. Commun. Image Represent..

[66]  Zhihui Wang,et al.  A multi-faceted adaptive image fusion algorithm using a multi-wavelet-based matching measure in the PCNN domain , 2017, Appl. Soft Comput..

[67]  Alexander Akerman,et al.  Pyramidal techniques for multisensor fusion , 1992, Other Conferences.

[68]  Harris Drucker,et al.  Learning algorithms for classification: A comparison on handwritten digit recognition , 1995 .

[69]  Abdul Majid,et al.  Random forest-based scheme using feature and decision levels information for multi-focus image fusion , 2015, Pattern Analysis and Applications.

[70]  Jin Li,et al.  Image Fusion Algorithm Based on Simplified PCNN in Nonsubsampled Contourlet Transform Domain , 2012 .

[71]  Yoshua Bengio,et al.  Extracting and composing robust features with denoising autoencoders , 2008, ICML '08.

[72]  Miroslav Kubat,et al.  Neural networks: a comprehensive foundation by Simon Haykin, Macmillan, 1994, ISBN 0-02-352781-7. , 1999, The Knowledge Engineering Review.

[73]  Wenzhong Shi,et al.  Multisource Image Fusion Method Using Support Value Transform , 2007, IEEE Transactions on Image Processing.

[74]  Yong Jiang,et al.  Image fusion using multiscale edge-preserving decomposition based on weighted least squares filter , 2014, IET Image Process..

[75]  Hongyi Liu,et al.  A New Pan-Sharpening Method With Deep Neural Networks , 2015, IEEE Geoscience and Remote Sensing Letters.

[76]  Abdul Majid,et al.  A novel ensemble scheme for the multi-focus image fusion using support vector machine , 2014, Int. J. Comput. Math..

[77]  Shan Suthaharan,et al.  Support Vector Machine , 2016 .

[78]  Jian Sun,et al.  Guided Image Filtering , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[79]  A. M. Turing,et al.  Computing Machinery and Intelligence , 1950, The Philosophy of Artificial Intelligence.

[80]  Dheeraj Agrawal,et al.  Multifocus image fusion using modified pulse coupled neural network for improved image quality , 2010 .

[81]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[82]  J. Zhou,et al.  A wavelet transform method to merge Landsat TM and SPOT panchromatic data , 1998 .

[83]  Sana Shaikh,et al.  Multi-focus Image Fusion Using Deep Belief Network , 2017 .

[84]  Shesheng Gao,et al.  Image Segmentation-Based Multi-Focus Image Fusion Through Multi-Scale Convolutional Neural Network , 2017, IEEE Access.

[85]  John McCarthy,et al.  A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence, August 31, 1955 , 2006, AI Mag..