SESV: Accurate Medical Image Segmentation by Predicting and Correcting Errors

Medical image segmentation is an essential task in computer-aided diagnosis. Despite their prevalence and success, deep convolutional neural networks (DCNNs) still need to be improved to produce accurate and robust enough segmentation results for clinical use. In this paper, we propose a novel and generic framework called Segmentation-Emendation-reSegmentation-Verification (SESV) to improve the accuracy of existing DCNNs in medical image segmentation, instead of designing a more accurate segmentation model. Our idea is to predict the segmentation errors produced by an existing model and then correct them. Since predicting segmentation errors is challenging, we design two ways to tolerate the mistakes in the error prediction. First, rather than using a predicted segmentation error map to correct the segmentation mask directly, we only treat the error map as the prior that indicates the locations where segmentation errors are prone to occur, and then concatenate the error map with the image and segmentation mask as the input of a re-segmentation network. Second, we introduce a verification network to determine whether to accept or reject the refined mask produced by the re-segmentation network on a region-by-region basis. The experimental results on the CRAG, ISIC, and IDRiD datasets suggest that using our SESV framework can improve the accuracy of DeepLabv3+ substantially and achieve advanced performance in the segmentation of gland cells, skin lesions, and retinal microaneurysms. Consistent conclusions can also be drawn when using PSPNet, U-Net, and FPN as the segmentation network, respectively. Therefore, our SESV framework is capable of improving the accuracy of different DCNNs on different medical image segmentation tasks.

[1]  Deniz Erdogmus,et al.  Auto-Context Convolutional Neural Network (Auto-Net) for Brain Extraction in Magnetic Resonance Imaging , 2017, IEEE Transactions on Medical Imaging.

[2]  Dagan Feng,et al.  Dermoscopic Image Segmentation via Multistage Fully Convolutional Networks , 2017, IEEE Transactions on Biomedical Engineering.

[3]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[4]  Yang Li,et al.  Gland Instance Segmentation Using Deep Multichannel Neural Networks , 2016, IEEE Transactions on Biomedical Engineering.

[5]  Dagan Feng,et al.  Improving Skin Lesion Segmentation via Stacked Adversarial Learning , 2019, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[6]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[7]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[8]  Thomas Walter,et al.  Segmentation of Nuclei in Histopathology Images by Deep Regression of the Distance Map , 2019, IEEE Transactions on Medical Imaging.

[9]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Jerry L Prince,et al.  Current methods in medical image segmentation. , 2000, Annual review of biomedical engineering.

[11]  Hao Chen,et al.  DCAN: Deep Contour-Aware Networks for Accurate Gland Segmentation , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Hao Chen,et al.  Light-Weight Hybrid Convolutional Network for Liver Tumor Segmentation , 2019, IJCAI.

[13]  Nima Tajbakhsh,et al.  UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation , 2020, IEEE Transactions on Medical Imaging.

[14]  Enoch Peserico,et al.  Simpler, Faster, More Accurate Melanocytic Lesion Segmentation Through MEDS , 2014, IEEE Transactions on Biomedical Engineering.

[15]  Jun Zhang,et al.  Hierarchical Convolutional Neural Networks for Segmentation of Breast Tumors in MRI With Application to Radiogenomics , 2019, IEEE Transactions on Medical Imaging.

[16]  David Dagan Feng,et al.  Saliency-Based Lesion Segmentation Via Background Detection in Dermoscopic Images , 2017, IEEE Journal of Biomedical and Health Informatics.

[17]  Xiaogang Wang,et al.  Pyramid Scene Parsing Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  John Willian Branch,et al.  Automatic skin lesion segmentation on dermoscopic images by the means of superpixel merging , 2018, MICCAI.

[19]  Noel C. F. Codella,et al.  Skin lesion analysis toward melanoma detection: A challenge at the 2017 International symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC) , 2016, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[20]  Harald Kittler,et al.  Descriptor : The HAM 10000 dataset , a large collection of multi-source dermatoscopic images of common pigmented skin lesions , 2018 .

[21]  Ling Shao,et al.  Collaborative Learning of Semi-Supervised Segmentation and Classification for Medical Images , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Yading Yuan,et al.  Automatic Skin Lesion Segmentation Using Deep Fully Convolutional Networks With Jaccard Distance , 2017, IEEE Transactions on Medical Imaging.

[23]  Anindo Saha,et al.  Leveraging Adaptive Color Augmentation in Convolutional Neural Networks for Deep Skin Lesion Segmentation , 2020, 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI).

[24]  Petia Radeva,et al.  SLSDeep: Skin Lesion Segmentation Based on Dilated Residual and Pyramid Pooling Networks , 2018, MICCAI.

[25]  Luc Van Gool,et al.  Temporal Segment Networks: Towards Good Practices for Deep Action Recognition , 2016, ECCV.

[26]  Hao Chen,et al.  Unsupervised Bidirectional Cross-Modality Adaptation via Deeply Synergistic Image and Feature Alignment for Medical Image Segmentation , 2020, IEEE Transactions on Medical Imaging.

[27]  Chunhua Shen,et al.  A Mutual Bootstrapping Model for Automated Skin Lesion Segmentation and Classification , 2020, IEEE Transactions on Medical Imaging.

[28]  Xin Yang,et al.  Deep Attentive Features for Prostate Segmentation in 3D Transrectal Ultrasound , 2019, IEEE Transactions on Medical Imaging.

[29]  Hang Li,et al.  Dense Deconvolutional Network for Skin Lesion Segmentation , 2019, IEEE Journal of Biomedical and Health Informatics.

[30]  Chi-Wing Fu,et al.  Patch-Based Output Space Adversarial Learning for Joint Optic Disc and Cup Segmentation , 2019, IEEE Transactions on Medical Imaging.

[31]  Farida Cheriet,et al.  A Novel Weakly Supervised Multitask Architecture for Retinal Lesions Segmentation on Fundus Images , 2019, IEEE Transactions on Medical Imaging.

[32]  Kwang-Ting Cheng,et al.  Enabling a Single Deep Learning Model for Accurate Gland Instance Segmentation: A Shape-Aware Adversarial Learning Framework , 2020, IEEE Transactions on Medical Imaging.

[33]  George Papandreou,et al.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation , 2018, ECCV.

[34]  Kaiming He,et al.  Feature Pyramid Networks for Object Detection , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[36]  Lei Xing,et al.  Modified U-Net (mU-Net) With Incorporation of Object-Dependent High Level Features for Improved Liver and Liver-Tumor Segmentation in CT Images , 2019, IEEE Transactions on Medical Imaging.

[37]  Yading Yuan,et al.  Improving Dermoscopic Image Segmentation With Enhanced Convolutional-Deconvolutional Networks , 2017, IEEE Journal of Biomedical and Health Informatics.

[38]  Wei Liang,et al.  Microaneurysms segmentation with a U-Net based on recurrent residual convolutional neural network , 2019, Journal of medical imaging.

[39]  François Chollet,et al.  Xception: Deep Learning with Depthwise Separable Convolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Matthieu Cord,et al.  Addressing Failure Prediction by Learning Model Confidence , 2019, NeurIPS.

[41]  Sébastien Ourselin,et al.  DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Chi-Wing Fu,et al.  H-DenseUNet: Hybrid Densely Connected UNet for Liver and Tumor Segmentation From CT Volumes , 2018, IEEE Transactions on Medical Imaging.

[43]  Yang Song,et al.  3D APA-Net: 3D Adversarial Pyramid Anisotropic Convolutional Network for Prostate Segmentation in MR Images , 2020, IEEE Transactions on Medical Imaging.

[44]  Nassir Navab,et al.  Multi-scale Microaneurysms Segmentation Using Embedding Triplet Loss , 2019, MICCAI.

[45]  Noel C. F. Codella,et al.  Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC) , 2019, ArXiv.

[46]  Hao Chen,et al.  MILD‐Net: Minimal information loss dilated network for gland instance segmentation in colon histology images , 2018, Medical Image Anal..

[47]  Lequan Yu,et al.  MS-Net: Multi-Site Network for Improving Prostate Segmentation With Heterogeneous MRI Data , 2020, IEEE Transactions on Medical Imaging.

[48]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[49]  Qin Li,et al.  Retinopathy Online Challenge: Automatic Detection of Microaneurysms in Digital Color Fundus Photographs , 2010, IEEE Transactions on Medical Imaging.

[50]  U. Rajendra Acharya,et al.  Automated segmentation of exudates, haemorrhages, microaneurysms using single convolutional neural network , 2017, Inf. Sci..

[51]  Hao Lu,et al.  Deep Segmentation-Emendation Model for Gland Instance Segmentation , 2019, MICCAI.

[52]  Bálint Antal,et al.  An Ensemble-Based System for Microaneurysm Detection and Diabetic Retinopathy Grading , 2012, IEEE Transactions on Biomedical Engineering.

[53]  Xin Yang,et al.  Towards Automated Semantic Segmentation in Prenatal Volumetric Ultrasound , 2019, IEEE Transactions on Medical Imaging.

[54]  Dimitris N. Metaxas,et al.  Improving Nuclei/Gland Instance Segmentation in Histopathology Images by Full Resolution Neural Network and Spatial Constrained Loss , 2019, MICCAI.

[55]  Jing Qin,et al.  Skin lesion segmentation via generative adversarial networks with dual discriminators , 2020, Medical Image Anal..

[56]  Ghassan Hamarneh,et al.  Star Shape Prior in Fully Convolutional Networks for Skin Lesion Segmentation , 2018, MICCAI.

[57]  Hao Chen,et al.  3D deeply supervised network for automated segmentation of volumetric medical images , 2017, Medical Image Anal..

[58]  Joseph O. Deasy,et al.  Multiple Resolution Residually Connected Feature Streams for Automatic Lung Tumor Segmentation From CT Images , 2019, IEEE Transactions on Medical Imaging.

[59]  Fabrice Mériaudeau,et al.  Indian Diabetic Retinopathy Image Dataset (IDRiD): A Database for Diabetic Retinopathy Screening Research , 2018, Data.

[60]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.