Classification of Medical Images in the Biomedical Literature by Jointly Using Deep and Handcrafted Visual Features

The classification of medical images and illustrations from the biomedical literature is important for automated literature review, retrieval, and mining. Although deep learning is effective for large-scale image classification, it may not be the optimal choice for this task as there is only a small training dataset. We propose a combined deep and handcrafted visual feature (CDHVF) based algorithm that uses features learned by three fine-tuned and pretrained deep convolutional neural networks (DCNNs) and two handcrafted descriptors in a joint approach. We evaluated the CDHVF algorithm on the ImageCLEF 2016 Subfigure Classification dataset and it achieved an accuracy of 85.47%, which is higher than the best performance of other purely visual approaches listed in the challenge leaderboard. Our results indicate that handcrafted features complement the image representation learned by DCNNs on small training datasets and improve accuracy in certain medical image classification problems.

[1]  Heng Huang,et al.  Large Margin Local Estimate With Applications to Medical Image Classification , 2015, IEEE Transactions on Medical Imaging.

[2]  Henning Müller,et al.  Evaluating performance of biomedical image retrieval systems - An overview of the medical image retrieval task at ImageCLEF 2004-2013 , 2015, Comput. Medical Imaging Graph..

[3]  Ivan Laptev,et al.  Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[5]  Sven Koitka,et al.  Traditional Feature Engineering and Deep Learning Approaches at Medical Classification Task of ImageCLEF 2016 , 2016, CLEF.

[6]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[7]  Hao Chen,et al.  Standard Plane Localization in Fetal Ultrasound via Domain Transferred Deep Neural Networks , 2015, IEEE Journal of Biomedical and Health Informatics.

[8]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[9]  Jiajun Wu,et al.  Deep multiple instance learning for image classification and auto-annotation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Juergen Gall,et al.  A BoW-equivalent Recurrent Neural Network for Action Recognition , 2015, BMVC.

[11]  Theodore Kalamboukis,et al.  IPL at CLEF 2016 Medical Task , 2016, CLEF.

[12]  Yann LeCun,et al.  What is the best multi-stage architecture for object recognition? , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[13]  Yihong Gong,et al.  Linear spatial pyramid matching using sparse coding for image classification , 2009, CVPR.

[14]  Fabio A. González,et al.  Extracting Salient Brain Patterns for Imaging-Based Classification of Neurodegenerative Diseases , 2014, IEEE Transactions on Medical Imaging.

[15]  João Magalhães,et al.  NovaSearch at ImageCLEFmed 2016 Subfigure Classification Task , 2016, CLEF.

[16]  Yan Xu,et al.  Deep learning of feature representation with multiple instance learning for medical image analysis , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[17]  Ashnil Kumar,et al.  Subfigure and Multi-Label Classification using a Fine-Tuned Convolutional Neural Network , 2016, CLEF.

[18]  Andrew Zisserman,et al.  Return of the Devil in the Details: Delving Deep into Convolutional Nets , 2014, BMVC.

[19]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[20]  Marek R. Ogiela,et al.  Artificial intelligence structural imaging techniques in visual pattern analysis and medical data understanding , 2003, Pattern Recognit..

[21]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Vasileios Megalooikonomou,et al.  A Representation and Classification Scheme for Tree-Like Structures in Medical Images: Analyzing the Branching Pattern of Ductal Trees in X-ray Galactograms , 2009, IEEE Transactions on Medical Imaging.

[23]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[24]  Matti Pietikäinen,et al.  A comparative study of texture measures with classification based on featured distributions , 1996, Pattern Recognit..

[25]  R. Gray,et al.  Vector quantization , 1984, IEEE ASSP Magazine.

[26]  Guang-Zhong Yang,et al.  Deep Learning for Health Informatics , 2017, IEEE Journal of Biomedical and Health Informatics.

[27]  Andrea Vedaldi,et al.  MatConvNet: Convolutional Neural Networks for MATLAB , 2014, ACM Multimedia.

[28]  Yihong Gong,et al.  Locality-constrained Linear Coding for image classification , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[29]  Stefano Bromuri,et al.  Overview of the medical tasks in ImageCLEF 2016 , 2016 .

[30]  Yiannis S. Boutalis,et al.  Compact Composite Descriptors for Content Based Image Retrieval: Basics, Concepts, Tools , 2011 .

[31]  Pietro Perona,et al.  A Bayesian hierarchical model for learning natural scene categories , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[32]  R. Keys.  Cubic convolution interpolation for digital image processing , 1981 .

[33]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[34]  Petra Perner.  Image mining: issues, framework, a generic tool and its application to medical-image diagnosis , 2002 .

[35]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[36]  Yiannis S. Boutalis,et al.  FCTH: Fuzzy Color and Texture Histogram - A Low Level Feature for Accurate Image Retrieval , 2008, 2008 Ninth International Workshop on Image Analysis for Multimedia Interactive Services.

[37]  Qi Tian,et al.  Ieee Transactions on Image Processing Spatial Pooling of Heterogeneous Features for Image Classification , 2022 .

[38]  Jiwen Lu,et al.  Deep transfer metric learning , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Cordelia Schmid,et al.  Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[40]  Sameer Antani,et al.  Creating a classification of image types in the medical literature for visual categorization , 2012, Other Conferences.

[41]  Payel Ghosh,et al.  Review of medical image retrieval systems and future directions , 2011, 2011 24th International Symposium on Computer-Based Medical Systems (CBMS).

[42]  Christoph M. Friedrich,et al.  FHDO Biomedical Computer Science Group at Medical Classification Task of ImageCLEF 2015 , 2015, CLEF.