Collaborative Learning of Semi-Supervised Segmentation and Classification for Medical Images

Medical image analysis has two important research areas: disease grading and fine-grained lesion segmentation. Although the former problem often relies on the latter, the two are usually studied separately. Disease severity grading can be treated as a classification problem, which only requires image-level annotations, while the lesion segmentation requires stronger pixel-level annotations. However, pixel-wise data annotation for medical images is highly time-consuming and requires domain experts. In this paper, we propose a collaborative learning method to jointly improve the performance of disease grading and lesion segmentation by semi-supervised learning with an attention mechanism. Given a small set of pixel-level annotated data, a multi-lesion mask generation model first performs the traditional semantic segmentation task. Then, based on initially predicted lesion maps for large quantities of image-level annotated data, a lesion attentive disease grading model is designed to improve the severity classification accuracy. Meanwhile, the lesion attention model can refine the lesion maps using class-specific information to fine-tune the segmentation model in a semi-supervised manner. An adversarial architecture is also integrated for training. With extensive experiments on a representative medical problem called diabetic retinopathy (DR), we validate the effectiveness of our method and achieve consistent improvements over state-of-the-art methods on three public datasets.

[1]  Yaozong Gao,et al.  ASDNet: Attention Based Semi-supervised Deep Networks for Medical Image Segmentation , 2018, MICCAI.

[2]  Mert R. Sabuncu,et al.  Anatomical Priors in Convolutional Networks for Unsupervised Biomedical Segmentation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[3]  Ritse Mann,et al.  Automated soft tissue lesion detection and segmentation in digital mammography using a u-net deep learning network , 2018, ArXiv.

[4]  Xiao Liu,et al.  Attention Clusters: Purely Attention Based Local Feature Integration for Video Classification , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[5]  Tao Li,et al.  Lesion Detection and Grading of Diabetic Retinopathy via Two-Stages Deep Convolutional Neural Networks , 2017, MICCAI.

[6]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Ling Shao,et al.  Viewpoint-Aware Attentive Multi-view Inference for Vehicle Re-identification , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[8]  Ioannis A. Kakadiaris,et al.  Deep Imbalanced Attribute Classification using Visual Attention Aggregation , 2018, ECCV.

[9]  Xiaogang Wang,et al.  Zoom-in-Net: Deep Mining Lesions for Diabetic Retinopathy Detection , 2017, MICCAI.

[10]  Matthieu Cord,et al.  HybridNet: Classification and Reconstruction Cooperation for Semi-Supervised Learning , 2018, ECCV.

[11]  Zhe Gan,et al.  AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[12]  Frans Coenen,et al.  Convolutional Neural Networks for Diabetic Retinopathy , 2016, MIUA.

[13]  Farida Cheriet,et al.  Automatic Grading of Diabetic Retinopathy on a Public Database , 2015 .

[14]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[15]  Aaas News,et al.  Book Reviews , 1893, Buffalo Medical and Surgical Journal.

[16]  Farida Cheriet,et al.  Red Lesion Detection Using Dynamic Shape Features for Diabetic Retinopathy Screening , 2016, IEEE Transactions on Medical Imaging.

[17]  Lei Zhang,et al.  Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[18]  Kaiming He,et al.  Mask R-CNN , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[19]  Fadime Sener,et al.  Unsupervised Learning and Segmentation of Complex Activities from Video , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[20]  George Papandreou,et al.  MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[21]  Abhishek Verma,et al.  New Deep Neural Nets for Fine-Grained Diabetic Retinopathy Recognition on Hybrid Color Space , 2016, 2016 IEEE International Symposium on Multimedia (ISM).

[22]  Bian Wu,et al.  A Framework for Identifying Diabetic Retinopathy Based on Anti-noise Detection and Attention-Based Fusion , 2018, MICCAI.

[23]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[24]  Bo Wang,et al.  Deep Co-Training for Semi-Supervised Image Recognition , 2018, ECCV.

[25]  Ling Shao,et al.  Vehicle Re-Identification by Deep Hidden Multi-View Inference , 2018, IEEE Transactions on Image Processing.

[26]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Tat-Seng Chua,et al.  SCA-CNN: Spatial and Channel-Wise Attention in Convolutional Networks for Image Captioning , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Guy Cazuguel,et al.  FEEDBACK ON A PUBLICLY DISTRIBUTED IMAGE DATABASE: THE MESSIDOR DATABASE , 2014 .

[29]  Meindert Niemeijer,et al.  Evaluation of a computer-aided diagnosis system for diabetic retinopathy screening on public data. , 2011, Investigative ophthalmology & visual science.

[30]  Yang Zou,et al.  Domain Adaptation for Semantic Segmentation via Class-Balanced Self-Training , 2018, ArXiv.

[31]  Doina Precup,et al.  Exploring Uncertainty Measures in Deep Networks for Multiple Sclerosis Lesion Detection and Segmentation , 2018, MICCAI.

[32]  Fabrice Mériaudeau,et al.  Indian Diabetic Retinopathy Image Dataset (IDRiD): A Database for Diabetic Retinopathy Screening Research , 2018, Data.

[33]  Seunghoon Hong,et al.  Decoupled Deep Neural Network for Semi-supervised Semantic Segmentation , 2015, NIPS.

[34]  Gang Yu,et al.  Learning a Discriminative Feature Network for Semantic Segmentation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[35]  Yunchao Wei,et al.  Revisiting Dilated Convolution: A Simple Approach for Weakly- and Semi-Supervised Semantic Segmentation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[36]  Cornelis H. Slump,et al.  Quantitative Collateral Grading on CT Angiography in Patients with Acute Ischemic Stroke , 2017, CMMI/RAMBO/SWITCH@MICCAI.

[37]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[38]  Prachi Gharpure,et al.  Diabetic retinopathy detection using deep convolutional neural networks , 2016, 2016 International Conference on Computing, Analytics and Security Trends (CAST).

[39]  Ming-Hsuan Yang,et al.  Adversarial Learning for Semi-supervised Semantic Segmentation , 2018, BMVC.

[40]  Ralph R. Martin,et al.  Associating Inter-image Salient Instances for Weakly Supervised Semantic Segmentation , 2018, ECCV.

[41]  Bálint Antal,et al.  An Ensemble-Based System for Microaneurysm Detection and Diabetic Retinopathy Grading , 2012, IEEE Transactions on Biomedical Engineering.

[42]  Sim Heng Ong,et al.  Focus, Segment and Erase: An Efficient Network for Multi-label Brain Tumor Segmentation , 2018, ECCV.

[43]  Bram van Ginneken,et al.  Fast Convolutional Neural Network Training Using Selective Data Sampling: Application to Hemorrhage Detection in Color Fundus Images , 2016, IEEE Transactions on Medical Imaging.

[44]  Marek Kowal,et al.  Feature selection for breast cancer malignancy classification problem , 2010 .

[45]  Kaiming He,et al.  Focal Loss for Dense Object Detection , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[46]  Le Lu,et al.  DeepLesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning , 2018, Journal of medical imaging.

[47]  Philip H. S. Torr,et al.  Weakly- and Semi-Supervised Panoptic Segmentation , 2022 .

[48]  Eka Miranda,et al.  A survey of medical image classification techniques , 2016, 2016 International Conference on Information Management and Technology (ICIMTech).

[49]  Xiao Liu,et al.  Fine-Grained Video Categorization with Redundancy Reduction Attention , 2018, ECCV.

[50]  François Chollet,et al.  Xception: Deep Learning with Depthwise Separable Convolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Huimin Ma,et al.  Weakly-Supervised Semantic Segmentation by Iteratively Mining Common Object Features , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[52]  George Papandreou,et al.  Weakly-and Semi-Supervised Learning of a Deep Convolutional Network for Semantic Image Segmentation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[53]  Subhashini Venugopalan,et al.  Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. , 2016, JAMA.