Challenges and Recent Solutions for Image Segmentation in the Era of Deep Learning

Image segmentation has a key role in computer vision and image processing. Superiority of deep learning based segmentation techniques has been shown in various studies in the literature. However, there are challenging issues affecting performances of these methods. Therefore, in this paper, these challenges that are mostly related to architecture and training of deep neural networks are explained. In addition, the state-of-the-art solutions applied in the literature are presented to help researchers to design proper network architectures according to their problems and to be aware of possible challenging issues and recent solutions.

[1]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[2]  Hao Chen,et al.  3D Deeply Supervised Network for Automatic Liver Segmentation from CT Volumes , 2016, MICCAI.

[3]  Purang Abolmaesumi,et al.  A Unified Framework Integrating Recurrent Fully-Convolutional Networks and Optical Flow for Segmentation of the Left Ventricle in Echocardiography Data , 2018, DLMIA/ML-CDS@MICCAI.

[4]  Anton van den Hengel,et al.  Wider or Deeper: Revisiting the ResNet Model for Visual Recognition , 2016, Pattern Recognit..

[5]  Seyed-Ahmad Ahmadi,et al.  Hough-CNN: Deep learning for segmentation of deep brain regions in MRI and ultrasound , 2016, Comput. Vis. Image Underst..

[6]  Tammy Riklin-Raviv,et al.  Microscopy Cell Segmentation Via Convolutional LSTM Networks , 2018, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[7]  Seyed-Ahmad Ahmadi,et al.  Automatic Liver and Lesion Segmentation in CT Using Cascaded Fully Convolutional Neural Networks and 3D Conditional Random Fields , 2016, MICCAI.

[8]  Jian Sun,et al.  ExFuse: Enhancing Feature Fusion for Semantic Segmentation , 2018, ECCV.

[9]  Bernt Schiele,et al.  Simple Does It: Weakly Supervised Instance and Semantic Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Andreas K. Maier,et al.  Dilated Deeply Supervised Networks for Hippocampus Segmentation in MRI , 2019, Bildverarbeitung für die Medizin.

[11]  David J. Kriegman,et al.  Dense Volume-to-Volume Vascular Boundary Detection , 2016, MICCAI.

[12]  Yang Lei,et al.  Automatic MRI prostate segmentation using 3D deeply supervised FCN with concatenated atrous convolution , 2019, Medical Imaging.

[13]  Sebastian J. Schlecht,et al.  Automatic Liver and Tumor Segmentation of CT and MRI Volumes using Cascaded Fully Convolutional Neural Networks , 2017, ArXiv.

[14]  Song Guo,et al.  Deeply supervised neural network with short connections for retinal vessel segmentation , 2018, Int. J. Medical Informatics.

[15]  Mark A. Anastasio,et al.  Automatic microscopic cell counting by use of deeply-supervised density regression model , 2019, Medical Imaging.

[16]  Hao Chen,et al.  3D deeply supervised network for automated segmentation of volumetric medical images , 2017, Medical Image Anal..

[17]  Fred A. Hamprecht,et al.  Multi-modal Brain Tumor Segmentation using Deep Convolutional Neural Networks , 2014 .

[18]  Gustavo Carneiro,et al.  Deep Learning and Structured Prediction for the Segmentation of Mass in Mammograms , 2015, MICCAI.

[19]  Hang Chang,et al.  Unsupervised Transfer Learning via Multi-Scale Convolutional Sparse Coding for Biomedical Applications , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Konstantinos Kamnitsas,et al.  Efficient multi‐scale 3D CNN with fully connected CRF for accurate brain lesion segmentation , 2016, Medical Image Anal..

[21]  Sidan Du,et al.  Cerebral Micro-Bleed Detection Based on the Convolution Neural Network With Rank Based Average Pooling , 2017, IEEE Access.

[22]  David J. Hawkes,et al.  Deep residual networks for automatic segmentation of laparoscopic videos of the liver , 2017, Medical Imaging.

[23]  Ajay Mittal,et al.  An Efficient Variant of Fully-Convolutional Network for Segmenting Lung Fields from Chest Radiographs , 2018, Wirel. Pers. Commun..

[24]  Peter Corcoran,et al.  Deep Neural Network and Data Augmentation Methodology for off-axis iris segmentation in wearable headsets , 2019, Neural Networks.

[25]  Ronald M. Summers,et al.  Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning , 2016, IEEE Transactions on Medical Imaging.

[26]  Eduardo Valle,et al.  Knowledge transfer for melanoma screening with deep learning , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[27]  Alexander Binder,et al.  Analyzing Classifiers: Fisher Vectors and Deep Neural Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  R Saranya,et al.  Automated Melanoma Recognition in Dermoscopy Images via Very Deep Residual Networks , 2019 .

[30]  Yang Lei,et al.  Automated prostate segmentation of volumetric CT images using 3D deeply supervised dilated FCN , 2019, Medical Imaging: Image Processing.

[31]  Marcia Hon,et al.  Towards Alzheimer's disease classification through transfer learning , 2017, 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM).

[32]  Khaled Shaalan,et al.  Speech Recognition Using Deep Neural Networks: A Systematic Review , 2019, IEEE Access.

[33]  Kyong Joon Lee,et al.  Lung nodule segmentation with convolutional neural network trained by simple diameter information , 2018 .

[34]  Jian Sun,et al.  ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Lin Yang,et al.  SPDA: Superpixel-based Data Augmentation for Biomedical Image Segmentation , 2019, MIDL.

[36]  Md Zahangir Alom,et al.  Recurrent residual U-Net for medical image segmentation , 2019, Journal of medical imaging.

[37]  Hyo-Eun Kim,et al.  Self-Transfer Learning for Weakly Supervised Lesion Localization , 2016, MICCAI.

[38]  Andreas Uhl,et al.  Simple domain adaptation for cross-dataset analyses of brain MRI data , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[39]  Li Li,et al.  DeepCrack: A deep hierarchical feature learning architecture for crack segmentation , 2019, Neurocomputing.

[40]  Frédo Durand,et al.  Data augmentation using learned transforms for one-shot medical image segmentation , 2019, ArXiv.

[41]  Yu Liu,et al.  A review of semantic segmentation using deep neural networks , 2017, International Journal of Multimedia Information Retrieval.

[42]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Tian Liu,et al.  Deeply supervised 3D fully convolutional networks with group dilated convolution for automatic MRI prostate segmentation , 2019, Medical physics.

[44]  Luca Maria Gambardella,et al.  Deep Neural Networks Segment Neuronal Membranes in Electron Microscopy Images , 2012, NIPS.

[45]  Yoshua Bengio,et al.  How transferable are features in deep neural networks? , 2014, NIPS.

[46]  Klaus H. Maier-Hein,et al.  Deep MRI brain extraction: A 3D convolutional neural network for skull stripping , 2016, NeuroImage.

[47]  Marleen de Bruijne,et al.  Transfer learning for multi-center classification of chronic obstructive pulmonary disease , 2017 .

[48]  Lauge Sørensen,et al.  Transfer Learning for Multicenter Classification of Chronic Obstructive Pulmonary Disease , 2017, IEEE Journal of Biomedical and Health Informatics.

[49]  Syed Muhammad Anwar,et al.  Deep Learning in Medical Image Analysis , 2017 .

[50]  Farida Cheriet,et al.  A Novel Weakly Supervised Multitask Architecture for Retinal Lesions Segmentation on Fundus Images , 2019, IEEE Transactions on Medical Imaging.

[51]  Pheng-Ann Heng,et al.  Cascaded Fully Convolutional Networks for automatic prenatal ultrasound image segmentation , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[52]  Edward Y. Chang,et al.  Transfer representation learning for medical image analysis , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[53]  E. Sudheer Kumar,et al.  Medical Image Analysis Using Deep Learning: A Systematic Literature Review , 2019, Emerging Technologies in Computer Engineering: Microservices in Big Data Analytics.

[54]  Xiaochun Cao,et al.  Survey of recent progress in semantic image segmentation with CNNs , 2017, Science China Information Sciences.

[55]  Marcel van Herk,et al.  FAst Segmentation Through SURface Fairing (FASTSURF): A novel semi-automatic hippocampus segmentation method , 2019, PloS one.

[56]  Xinyue Zhao,et al.  Context-based conditional random fields as recurrent neural networks for image labeling , 2019, Multimedia Tools and Applications.

[57]  Guoyan Zheng,et al.  Multi-stream 3D FCN with multi-scale deep supervision for multi-modality isointense infant brain MR image segmentation , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[58]  Manoranjan Paul,et al.  Deep Learning Models for Retinal Blood Vessels Segmentation: A Review , 2019, IEEE Access.

[59]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[60]  Hao Chen,et al.  Deep Contextual Networks for Neuronal Structure Segmentation , 2016, AAAI.

[61]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[62]  Andries Engelbrecht,et al.  Loss Surface Modality of Feed-Forward Neural Network Architectures , 2019, 2020 International Joint Conference on Neural Networks (IJCNN).

[63]  Xindong Wu,et al.  Object Detection With Deep Learning: A Review , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[64]  Luiz Eduardo Soares de Oliveira,et al.  Deep features for breast cancer histopathological image classification , 2017, 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[65]  Elsa D. Angelini,et al.  Discriminative Localization in CNNs for Weakly-Supervised Segmentation of Pulmonary Nodules , 2017, MICCAI.

[66]  Amr Badr,et al.  Lung Cancer Detection and Classification with 3D Convolutional Neural Network (3D-CNN) , 2017 .

[67]  Mohamed S. Elmahdy,et al.  Low quality dermal image classification using transfer learning , 2017, 2017 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI).

[68]  Jian Sun,et al.  Identity Mappings in Deep Residual Networks , 2016, ECCV.

[69]  Zheng Lu,et al.  Survey of weakly supervised semantic segmentation methods , 2018, 2018 Chinese Control And Decision Conference (CCDC).

[70]  Hao Chen,et al.  Automated Melanoma Recognition in Dermoscopy Images via Very Deep Residual Networks , 2017, IEEE Transactions on Medical Imaging.

[71]  Quoc V. Le,et al.  AutoAugment: Learning Augmentation Policies from Data , 2018, ArXiv.

[72]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[73]  Marc Pollefeys,et al.  An Exploration of 2D and 3D Deep Learning Techniques for Cardiac MR Image Segmentation , 2017, STACOM@MICCAI.

[74]  Sinisa Todorovic,et al.  Combining Bottom-Up, Top-Down, and Smoothness Cues for Weakly Supervised Image Segmentation , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[75]  Mohammad H. Jafari,et al.  Dense Fully Convolutional Network for Skin Lesion Segmentation , 2017, ArXiv.

[76]  Fei-Fei Li,et al.  What's the Point: Semantic Segmentation with Point Supervision , 2015, ECCV.

[77]  Hyojin Kim,et al.  Lung nodule detection using 3D convolutional neural networks trained on weakly labeled data , 2016, SPIE Medical Imaging.

[78]  Ghassan Hamarneh,et al.  Deep features to classify skin lesions , 2016, 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI).

[79]  Ebrahim Nasr-Esfahani,et al.  Extraction of skin lesions from non-dermoscopic images for surgical excision of melanoma , 2017, International Journal of Computer Assisted Radiology and Surgery.

[80]  D. Louis Collins,et al.  The EADC-ADNI harmonized protocol for hippocampal segmentation: A validation study , 2018, NeuroImage.

[81]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[82]  Su-Lin Lee,et al.  Focal FCN: Towards Small Object Segmentation with Limited Training Data , 2017, ArXiv.

[83]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[84]  Chi-Wing Fu,et al.  Deeply Supervised Rotation Equivariant Network for Lesion Segmentation in Dermoscopy Images , 2018, OR 2.0/CARE/CLIP/ISIC@MICCAI.

[85]  Jian Sun,et al.  BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).