Medical image retrieval using deep convolutional neural network

Abstract With a widespread use of digital imaging data in hospitals, the size of medical image repositories is increasing rapidly. This causes difficulty in managing and querying these large databases leading to the need of content based medical image retrieval (CBMIR) systems. A major challenge in CBMIR systems is the semantic gap that exists between the low level visual information captured by imaging devices and high level semantic information perceived by human. The efficacy of such systems is more crucial in terms of feature representations that can characterize the high-level information completely. In this paper, we propose a framework of deep learning for CBMIR system by using deep convolutional neural network (CNN) that is trained for classification of medical images. An intermodal dataset that contains twenty-four classes and five modalities is used to train the network. The learned features and the classification results are used to retrieve medical images. For retrieval, best results are achieved when class based predictions are used. An average classification accuracy of 99.77% and a mean average precision of 0.69 is achieved for retrieval task. The proposed method is best suited to retrieve multimodal medical images for different body organs.

[1]  John P. Eakins,et al.  Towards intelligent image retrieval , 2002, Pattern Recognit..

[2]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[3]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[4]  S. Sathiamoorthy,et al.  A unified learning framework for content based medical image retrieval using a statistical model , 2016, J. King Saud Univ. Comput. Inf. Sci..

[5]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[6]  G. Wiselin Jiji,et al.  Content-based image retrieval in dermatology using intelligent technique , 2015, IET Image Process..

[7]  Ji Wan,et al.  Deep Learning for Content-Based Image Retrieval: A Comprehensive Study , 2014, ACM Multimedia.

[8]  Xiang Zhang,et al.  OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks , 2013, ICLR.

[9]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[10]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[11]  Ronald M. Summers,et al.  Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning , 2016, IEEE Transactions on Medical Imaging.

[12]  Haejun Lee,et al.  Medical Image Retrieval: Past and Present , 2012, Healthcare informatics research.

[13]  Luc Van Gool,et al.  SURF: Speeded Up Robust Features , 2006, ECCV.

[14]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups , 2012, IEEE Signal Processing Magazine.

[15]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[16]  Cordelia Schmid,et al.  Aggregating Local Image Descriptors into Compact Codes , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Pascal Vincent,et al.  Unsupervised Feature Learning and Deep Learning: A Review and New Perspectives , 2012, ArXiv.

[18]  Alexei A. Efros,et al.  Discovering objects and their location in images , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[19]  Chong-Wah Ngo,et al.  Evaluating bag-of-visual-words representations in scene classification , 2007, MIR '07.

[20]  Nitish Srivastava,et al.  Improving neural networks by preventing co-adaptation of feature detectors , 2012, ArXiv.

[21]  Fei-Fei Li,et al.  Large-Scale Video Classification with Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Wenjie Lu,et al.  Regional deep learning model for visual tracking , 2016, Neurocomputing.

[24]  Marleen de Bruijne,et al.  Combining Generative and Discriminative Representation Learning for Lung CT Analysis With Convolutional Restricted Boltzmann Machines , 2016, IEEE Transactions on Medical Imaging.

[25]  R. Balasubramanian,et al.  Local maximum edge binary patterns: A new descriptor for image retrieval and object tracking , 2012, Signal Process..

[26]  Mansour Jamzad,et al.  Content based image retrieval using the knowledge of texture, color and binary tree structure , 2009, 2009 Canadian Conference on Electrical and Computer Engineering.

[27]  Alekseĭ Grigorʹevich Ivakhnenko,et al.  CYBERNETIC PREDICTING DEVICES , 1966 .

[28]  Mei Chen,et al.  Dictionary pruning with visual word significance for medical image retrieval , 2016, Neurocomputing.

[29]  Lei Wu,et al.  Enhancing Bag-of-Words Models with Semantics-Preserving Metric Learning , 2011, IEEE MultiMedia.

[30]  S. Santhosh Baboo,et al.  Image Retrieval using Harris Corners and Histogram of Oriented Gradients , 2011 .

[31]  B. S. Manjunath,et al.  Texture Features for Browsing and Retrieval of Image Data , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  Ying Liu,et al.  A survey of content-based image retrieval with high-level semantics , 2007, Pattern Recognit..

[33]  Max A. Viergever,et al.  Automatic Segmentation of MR Brain Images With a Convolutional Neural Network , 2016, IEEE Transactions on Medical Imaging.

[34]  K. Kranthi Kumar,et al.  A novel approach to self order feature reweighting in CBIR to reduce semantic gap using Relevance Feedback , 2014, 2014 International Conference on Circuits, Power and Computing Technologies [ICCPCT-2014].

[35]  Shi Gang,et al.  Application of Image SIFT Features to the Context of CBIR , 2008, 2008 International Conference on Computer Science and Software Engineering.

[36]  Hamid R. Tizhoosh,et al.  Autoencoding the retrieval relevance of medical images , 2015, 2015 International Conference on Image Processing Theory, Tools and Applications (IPTA).

[37]  Shu Liao,et al.  Multi-Instance Deep Learning: Discover Discriminative Local Anatomies for Bodypart Recognition , 2016, IEEE Transactions on Medical Imaging.

[38]  Marios Anthimopoulos,et al.  Lung Pattern Classification for Interstitial Lung Diseases Using a Deep Convolutional Neural Network , 2016, IEEE Transactions on Medical Imaging.

[39]  Richard Hans Robert Hahnloser,et al.  Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit , 2000, Nature.

[40]  Guy Cazuguel,et al.  FEEDBACK ON A PUBLICLY DISTRIBUTED IMAGE DATABASE: THE MESSIDOR DATABASE , 2014 .

[41]  Michèle Allard,et al.  Feature-based brain MRI retrieval for Alzheimer disease diagnosis , 2012, 2012 19th IEEE International Conference on Image Processing.

[42]  Anil K. Jain,et al.  Image retrieval using color and shape , 1996, Pattern Recognit..

[43]  Noramiza Hashim,et al.  Content Based Image Retrieval and Classification using speeded-up robust features (SURF) and grouped bag-of-visual-words (GBoVW) , 2013, 2013 International Conference on Technology, Informatics, Management, Engineering and Environment.

[44]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[45]  Gwénolé Quellec,et al.  Fast Wavelet-Based Image Characterization for Highly Adaptive Image Retrieval , 2012, IEEE Transactions on Image Processing.

[46]  Jen-Hao Hsiao,et al.  Deep learning of binary hash codes for fast image retrieval , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[47]  Xiaolong Wang,et al.  Active deep learning method for semi-supervised sentiment classification , 2013, Neurocomputing.

[48]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[49]  George R. Thoma,et al.  A Learning-Based Similarity Fusion and Filtering Approach for Biomedical Image Retrieval Using SVM Classification and Relevance Feedback , 2011, IEEE Transactions on Information Technology in Biomedicine.

[50]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[51]  Victor S. Lempitsky,et al.  Aggregating Local Deep Features for Image Retrieval , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[52]  David J. Fleet,et al.  Hamming Distance Metric Learning , 2012, NIPS.

[53]  Yongdong Zhang,et al.  Scalable Similarity Search With Topology Preserving Hashing , 2014, IEEE Transactions on Image Processing.