Medical image classification using synergic deep learning

&NA; The classification of medical images is an essential task in computer‐aided diagnosis, medical image retrieval and mining. Although deep learning has shown proven advantages over traditional methods that rely on the handcrafted features, it remains challenging due to the significant intra‐class variation and inter‐class similarity caused by the diversity of imaging modalities and clinical pathologies. In this paper, we propose a synergic deep learning (SDL) model to address this issue by using multiple deep convolutional neural networks (DCNNs) simultaneously and enabling them to mutually learn from each other. Each pair of DCNNs has their learned image representation concatenated as the input of a synergic network, which has a fully connected structure that predicts whether the pair of input images belong to the same class. Thus, if one DCNN makes a correct classification, a mistake made by the other DCNN leads to a synergic error that serves as an extra force to update the model. This model can be trained end‐to‐end under the supervision of classification errors from DCNNs and synergic errors from each pair of DCNNs. Our experimental results on the ImageCLEF‐2015, ImageCLEF‐2016, ISIC‐2016, and ISIC‐2017 datasets indicate that the proposed SDL model achieves the state‐of‐the‐art performance in these medical image classification tasks.

[1]  Yanning Zhang,et al.  Fusing texture, shape and deep model-learned information at decision level for automated classification of lung nodules on chest CT , 2018, Inf. Fusion.

[2]  David Dagan Feng,et al.  Classification of Medical Images in the Biomedical Literature by Jointly Using Deep and Handcrafted Visual Features , 2018, IEEE Journal of Biomedical and Health Informatics.

[3]  Nasir M. Rajpoot,et al.  Locality Sensitive Deep Learning for Detection and Classification of Nuclei in Routine Colon Cancer Histology Images , 2016, IEEE Trans. Medical Imaging.

[4]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[5]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[6]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Marleen de Bruijne,et al.  Machine learning approaches in medical image analysis: From detection to diagnosis , 2016, Medical Image Anal..

[8]  V. Matousek,et al.  Signature verification using ART-2 neural network , 2002, Proceedings of the 9th International Conference on Neural Information Processing, 2002. ICONIP '02..

[9]  Karen Livescu,et al.  Deep convolutional acoustic word embeddings using word-pair side information , 2015, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[10]  S. Thrun,et al.  Corrigendum: Dermatologist-level classification of skin cancer with deep neural networks , 2017, Nature.

[11]  David Dagan Feng,et al.  Feature-Based Image Patch Approximation for Lung Tissue Classification , 2013, IEEE Transactions on Medical Imaging.

[12]  N. Rajpoot,et al.  Locality Sensitive Deep Learning for Detection and Classification of Nuclei in Routine Colon Cancer Histology Images , 2016, IEEE Transactions on Medical Imaging.

[13]  Yang Li,et al.  Melanoma Classification on Dermoscopy Images Using a Neural Network Ensemble Model , 2017, IEEE Transactions on Medical Imaging.

[14]  Yann LeCun,et al.  Signature Verification Using A "Siamese" Time Delay Neural Network , 1993, Int. J. Pattern Recognit. Artif. Intell..

[15]  Hamid Krim,et al.  Flexible Skew-Symmetric Shape Model for Shape Representation, Classification, and Sampling , 2007, IEEE Transactions on Image Processing.

[16]  Jason Weston,et al.  Curriculum learning , 2009, ICML '09.

[17]  Noel C. F. Codella,et al.  Skin lesion analysis toward melanoma detection: A challenge at the 2017 International symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC) , 2016, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[18]  Ivan Laptev,et al.  Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Alla Eddine Guissous.  Skin Lesion Classification Using Deep Neural Network , 2019, ArXiv.

[20]  Yi Su,et al.  A Novel Multi-task Deep Learning Model for Skin Lesion Segmentation and Classification , 2017, ArXiv.

[21]  Guang Yang,et al.  Automatic Brain Tumor Detection and Segmentation Using U-Net Based Fully Convolutional Networks , 2017, MIUA.

[22]  Hao Chen,et al.  DCAN: Deep contour‐aware networks for object instance segmentation from histology images , 2017, Medical Image Anal..

[23]  Guang Yang,et al.  MRI Brain Tumor Segmentation and Patient Survival Prediction Using Random Forests and Fully Convolutional Networks , 2017, BrainLes@MICCAI.

[24]  Yann LeCun,et al.  Learning a similarity metric discriminatively, with application to face verification , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[25]  Payel Ghosh,et al.  Review of medical image retrieval systems and future directions , 2011, 2011 24th International Symposium on Computer-Based Medical Systems (CBMS).

[26]  Sameer Antani,et al.  Creating a classification of image types in the medical literature for visual categorization , 2012, Other Conferences.

[27]  Theodore Kalamboukis,et al.  IPL at CLEF 2016 Medical Task , 2016, CLEF.

[28]  Iván González-Díaz,et al.  Incorporating the Knowledge of Dermatologists to Convolutional Neural Networks for the Diagnosis of Skin Lesions , 2017, ArXiv.

[29]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Wei Shen,et al.  Multi-crop Convolutional Neural Networks for lung nodule malignancy suspiciousness classification , 2017, Pattern Recognit..

[31]  Henning Müller,et al.  Overview of the ImageCLEF 2013 Medical Tasks , 2013, CLEF.

[32]  Aren Jansen,et al.  A comparison of neural network methods for unsupervised representation learning on the zero resource speech challenge , 2015, INTERSPEECH.

[33]  Christoph M. Friedrich,et al.  FHDO Biomedical Computer Science Group at Medical Classification Task of ImageCLEF 2015 , 2015, CLEF.

[34]  Jonas Mueller,et al.  Siamese Recurrent Architectures for Learning Sentence Similarity , 2016, AAAI.

[35]  L. Personnaz,et al.  Collective computational properties of neural networks: New learning mechanisms. , 1986, Physical review. A, General physics.

[36]  Lei Zhang,et al.  Fine-Tuning Convolutional Neural Networks for Biomedical Image Analysis: Actively and Incrementally , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Yan Xu,et al.  Deep learning of feature representation with multiple instance learning for medical image analysis , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[38]  Sebastian Thrun,et al.  Dermatologist-level classification of skin cancer with deep neural networks , 2017, Nature.

[39]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[40]  Pol Cirujeda,et al.  Medical Image Classification via 2D color feature based Covariance Descriptors , 2015, CLEF.

[41]  Hao Chen,et al.  Automated Melanoma Recognition in Dermoscopy Images via Very Deep Residual Networks , 2017, IEEE Transactions on Medical Imaging.

[42]  David Dagan Feng,et al.  Automatic Skin Lesion Analysis using Large-scale Dermoscopy Images and Deep Residual Networks , 2017, ArXiv.

[43]  Qi Wu,et al.  Skin Lesion Classification in Dermoscopy Images Using Synergic Deep Learning , 2018, MICCAI.

[44]  Hongfei Lin,et al.  Deep Transfer Learning for Modality Classification of Medical Images , 2017, Inf..

[45]  Heng Huang,et al.  Large Margin Local Estimate With Applications to Medical Image Classification , 2015, IEEE Transactions on Medical Imaging.

[46]  Henning Müller,et al.  Evaluating performance of biomedical image retrieval systems - An overview of the medical image retrieval task at ImageCLEF 2004-2013 , 2015, Comput. Medical Imaging Graph..

[47]  Aren Jansen,et al.  Unsupervised neural network based feature extraction using weak top-down constraints , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[48]  Jürgen Weese,et al.  Four challenges in medical image analysis from an industrial perspective , 2016, Medical Image Anal..

[49]  Hariharan Ravishankar,et al.  Learning and Incorporating Shape Models for Semantic Segmentation , 2017, MICCAI.

[50]  Sven Koitka,et al.  Traditional Feature Engineering and Deep Learning Approaches at Medical Classification Task of ImageCLEF 2016 , 2016, CLEF.

[51]  David Dagan Feng,et al.  An Ensemble of Fine-Tuned Convolutional Neural Networks for Medical Image Classification , 2017, IEEE Journal of Biomedical and Health Informatics.

[52]  Eduardo Valle,et al.  RECOD Titans at ISIC Challenge 2017 , 2017, ArXiv.

[53]  Shuiwang Ji,et al.  Deep Learning Segmentation of Optical Microscopy Images Improves 3-D Neuron Reconstruction , 2017, IEEE Transactions on Medical Imaging.

[54]  Henning Müller,et al.  Overview of the ImageCLEF 2015 Medical Classification Task , 2015, CLEF.

[55]  Dennis Koelma,et al.  The ImageNet Shuffle: Reorganized Pre-training for Video Event Detection , 2016, ICMR.