A survey on U-shaped networks in medical image segmentations

Abstract The U-shaped network is one of the end-to-end convolutional neural networks (CNNs). In electron microscope segmentation of ISBI challenge 2012, the concise architecture and outstanding performance of the U-shaped network are impressive. Then, a variety of segmentation models based on this architecture have been proposed for medical image segmentations. We present a comprehensive literature review of U-shaped networks applied to medical image segmentation tasks, focusing on the architectures, extended mechanisms and application areas in these studies. The aim of this survey is twofold. First, we report the different extended U-shaped networks, discuss main state-of-the-art extended mechanisms, including residual mechanism, dense mechanism, dilated mechanism, attention mechanism, multi-module mechanism, and ensemble mechanism, analyze their pros and cons. Second, this survey provides the overview of studies in main application areas of U-shaped networks, including brain tumor, stroke, white matter hyperintensities (WMHs), eye, cardiac, liver, musculoskeletal, skin cancer, and neuronal pathology. Finally, we summarize the current U-shaped networks, point out the open challenges and directions for future research.

[1]  Adam P. Harrison,et al.  White matter hyperintensity segmentation from T1 and FLAIR images using fully convolutional neural networks enhanced with residual connections , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[2]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[3]  Manabu Inoue,et al.  Early Diffusion-Weighted Imaging and Perfusion-Weighted Imaging Lesion Volumes Forecast Final Infarct Size in DEFUSE 2 , 2013, Stroke.

[4]  Nils Daniel Forkert,et al.  Multivariate Dynamic Prediction of Ischemic Infarction and Tissue Salvage as a Function of Time and Degree of Recanalization , 2015, Journal of cerebral blood flow and metabolism : official journal of the International Society of Cerebral Blood Flow and Metabolism.

[5]  Guang Yang,et al.  Automatic Brain Tumor Detection and Segmentation Using U-Net Based Fully Convolutional Networks , 2017, MIUA.

[6]  Jundong Liu,et al.  Hippocampus segmentation through multi-view ensemble ConvNets , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[7]  Demetri Terzopoulos,et al.  Deformable models in medical image analysis: a survey , 1996, Medical Image Anal..

[8]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[9]  Vijayan K. Asari,et al.  Nuclei Segmentation with Recurrent Residual Convolutional Neural Networks based U-Net (R2U-Net) , 2018, NAECON 2018 - IEEE National Aerospace and Electronics Conference.

[10]  Jose Dolz,et al.  Dense Multi-path U-Net for Ischemic Stroke Lesion Segmentation in Multiple Image Modalities , 2018, BrainLes@MICCAI.

[11]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[12]  et al.,et al.  ISLES 2015 ‐ A public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI , 2017, Medical Image Anal..

[13]  Pablo Lamata,et al.  Recurrent Fully Convolutional Neural Networks for Multi-slice MRI Cardiac Segmentation , 2016, RAMBO+HVSMR@MICCAI.

[14]  Yeonggul Jang,et al.  Automatic Segmentation of LV and RV in Cardiac MRI , 2017, STACOM@MICCAI.

[15]  Yue Zhang,et al.  A post-processing method to improve the white matter hyperintensity segmentation accuracy for randomly-initialized U-net , 2018, 2018 IEEE 23rd International Conference on Digital Signal Processing (DSP).

[16]  Ge Wang,et al.  A Perspective on Deep Imaging , 2016, IEEE Access.

[17]  Wiro J. Niessen,et al.  White matter lesion extension to automatic brain tissue segmentation on MRI , 2009, NeuroImage.

[18]  Shuiwang Ji,et al.  Deep convolutional neural networks for multi-modality isointense infant brain image segmentation , 2015, NeuroImage.

[19]  Jong Chul Ye,et al.  Framing U-Net via Deep Convolutional Framelets: Application to Sparse-View CT , 2017, IEEE Transactions on Medical Imaging.

[20]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[21]  Nicholas J. Tustison,et al.  Brain Tumor Segmentation Using an Ensemble of 3D U-Nets and Overall Survival Prediction Using Radiomic Features , 2018, Frontiers in Computational Neuroscience.

[22]  Noel C. F. Codella,et al.  Skin lesion analysis toward melanoma detection: A challenge at the 2017 International symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC) , 2016, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[23]  Paolo Perrotta,et al.  Automatic Detection of White Matter Hyperintensities in Healthy Aging and Pathology Using Magnetic Resonance Imaging: A Review , 2015, Neuroinformatics.

[24]  Jose Dolz,et al.  IVD-Net: Intervertebral disc localization and segmentation in MRI with a multi-modal UNet , 2018, CSI@MICCAI.

[25]  Tarek Khadir,et al.  Deep Convolutional Neural Networks Using U-Net for Automatic Brain Tumor Segmentation in Multimodal MRI Volumes , 2018, BrainLes@MICCAI.

[26]  Hao Chen,et al.  3D deeply supervised network for automated segmentation of volumetric medical images , 2017, Medical Image Anal..

[27]  Antonio Jose Rodríguez-Sánchez,et al.  ISLES Challenge: U-Shaped Convolution Neural Network with Dilated Convolution for 3D Stroke Lesion Segmentation , 2018, BrainLes@MICCAI.

[28]  Liu Jin,et al.  A survey of MRI-based brain tumor segmentation methods , 2014 .

[29]  Xin Yang,et al.  Class-Balanced Deep Neural Network for Automatic Ventricular Structure Segmentation , 2017, STACOM@MICCAI.

[30]  James M. Brown,et al.  Automatic assessment of glioma burden: a deep learning algorithm for fully automated volumetric and bidimensional measurement , 2019, Neuro-oncology.

[31]  Fei-Fei Li,et al.  Deep visual-semantic alignments for generating image descriptions , 2015, CVPR.

[32]  Hao Chen,et al.  Volumetric ConvNets with Mixed Residual Connections for Automated Prostate Segmentation from 3D MR Images , 2017, AAAI.

[33]  Raghav Mehta,et al.  M-net: A Convolutional Neural Network for deep brain structure segmentation , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[34]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2015, IEEE Trans. Pattern Anal. Mach. Intell..

[35]  Li Wang,et al.  STRAINet: Spatially Varying sTochastic Residual AdversarIal Networks for MRI Pelvic Organ Segmentation , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[36]  Mazhar Shaikh,et al.  Brain Tumor Segmentation Using Dense Fully Convolutional Neural Network , 2017, BrainLes@MICCAI.

[37]  Dhanesh Ramachandram,et al.  Automatic white matter lesion segmentation using an adaptive outlier detection method. , 2012, Magnetic resonance imaging.

[38]  Andriy Myronenko,et al.  3D MRI brain tumor segmentation using autoencoder regularization , 2018, BrainLes@MICCAI.

[39]  Joong-Ho Won,et al.  Ensemble of Deep Convolutional Neural Networks for Prognosis of Ischemic Stroke , 2016, BrainLes@MICCAI.

[40]  Wei-Shi Zheng,et al.  Fully convolutional network ensembles for white matter hyperintensities segmentation in MR images , 2018, NeuroImage.

[41]  Jean Stawiaski,et al.  A Pretrained DenseNet Encoder for Brain Tumor Segmentation , 2018, BrainLes@MICCAI.

[42]  Syed Muhammad Anwar,et al.  Segmentation of Glioma Tumors in Brain Using Deep Convolutional Neural Network , 2017, Neurocomputing.

[43]  Kunihiko Fukushima,et al.  Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position , 1980, Biological Cybernetics.

[44]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[45]  Alex Graves,et al.  Recurrent Models of Visual Attention , 2014, NIPS.

[46]  Bjoern H. Menze,et al.  Automatic Brain Structures Segmentation Using Deep Residual Dilated U-Net , 2018, BrainLes@MICCAI.

[47]  Heung-Il Suk,et al.  Deep Learning in Medical Image Analysis. , 2017, Annual review of biomedical engineering.

[48]  Xiaochun Cao,et al.  Joint Optic Disc and Cup Segmentation Based on Multi-Label Deep Network and Polar Transformation , 2018, IEEE Transactions on Medical Imaging.

[49]  D. Shepard,et al.  Computer Mapping: The SYMAP Interpolation Algorithm , 1984 .

[50]  Daniël M Pelt,et al.  A mixed-scale dense convolutional neural network for image analysis , 2017, Proceedings of the National Academy of Sciences.

[51]  Derek C. Rose,et al.  Deep Machine Learning - A New Frontier in Artificial Intelligence Research [Research Frontier] , 2010, IEEE Computational Intelligence Magazine.

[52]  Xiaofeng Zhu,et al.  Deep convolutional neural network for accurate segmentation and quantification of white matter hyperintensities , 2020, Neurocomputing.

[53]  S. Sk.  A Survey of MRI-Based Brain Tumor Segmentation Methods , 2014 .

[54]  Konstantinos Kamnitsas,et al.  Efficient multi‐scale 3D CNN with fully connected CRF for accurate brain lesion segmentation , 2016, Medical Image Anal..

[55]  Johan H. C. Reiber,et al.  Fully automatic segmentation of white matter hyperintensities in MR images of the elderly , 2005, NeuroImage.

[56]  Nassir Navab,et al.  InfiNet: Fully convolutional networks for infant brain MRI segmentation , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[57]  J. Pluim,et al.  Evaluation of a deep learning approach for the segmentation of brain tissues and white matter hyperintensities of presumed vascular origin in MRI , 2017, NeuroImage: Clinical.

[58]  Florian Jung,et al.  Evaluation of prostate segmentation algorithms for MRI: The PROMISE12 challenge , 2014, Medical Image Anal..

[59]  Phil Blunsom,et al.  Teaching Machines to Read and Comprehend , 2015, NIPS.

[60]  Toan Duc Bui,et al.  A Skip-Connected 3D DenseNet Networks with Adversarial Training for Volumetric Segmentation , 2018, BrainLes@MICCAI.

[61]  Jianxin Wang,et al.  Efficient multi-kernel DCNN with pixel dropout for stroke MRI segmentation , 2019, Neurocomputing.

[62]  Kun Zhang,et al.  Multi-scale Masked 3-D U-Net for Brain Tumor Segmentation , 2018, BrainLes@MICCAI.

[63]  Shubham Jain,et al.  2D-3D Fully Convolutional Neural Networks for Cardiac MR Segmentation , 2017, STACOM@MICCAI.

[64]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[65]  Ghassan Hamarneh,et al.  Deep features to classify skin lesions , 2016, 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI).

[66]  Klaus H. Maier-Hein,et al.  Statistical Atlases and Computational Models of the Heart. ACDC and MMWHS Challenges , 2017, Lecture Notes in Computer Science.

[67]  Xiaohui Xie,et al.  AnatomyNet: Deep learning for fast and fully automated whole‐volume segmentation of head and neck anatomy , 2018, Medical physics.

[68]  C. Jack,et al.  FLAIR histogram segmentation for measurement of leukoaraiosis volume , 2001, Journal of magnetic resonance imaging : JMRI.

[69]  Jose Dolz,et al.  3D fully convolutional networks for subcortical segmentation in MRI: A large-scale study , 2016, NeuroImage.

[70]  Ji Won Han,et al.  Application of variable threshold intensity to segmentation for white matter hyperintensities in fluid attenuated inversion recovery magnetic resonance images , 2014, Neuroradiology.

[71]  Sang Hyun Park,et al.  3D Patchwise U-Net with Transition Layers for MR Brain Segmentation , 2018, BrainLes@MICCAI.

[72]  Tien Yin Wong,et al.  ORIGA-light: An online retinal fundus image database for glaucoma analysis and research , 2010, 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology.

[73]  Rita Simões,et al.  Automatic segmentation of cerebral white matter hyperintensities using only 3D FLAIR images. , 2013, Magnetic resonance imaging.

[74]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[75]  Thomas Brox,et al.  3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation , 2016, MICCAI.

[76]  Jian Chen,et al.  Development and validation of morphological segmentation of age-related cerebral white matter hyperintensities , 2009, NeuroImage.

[77]  Jun Zhang,et al.  Hierarchical Convolutional Neural Networks for Segmentation of Breast Tumors in MRI With Application to Radiogenomics , 2019, IEEE Transactions on Medical Imaging.

[78]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[79]  Jianxin Wang,et al.  Multi-scale Deep Convolutional Neural Network for Stroke Lesions Segmentation on CT Images , 2018, BrainLes@MICCAI.

[80]  Kenji Suzuki,et al.  Overview of deep learning in medical imaging , 2017, Radiological Physics and Technology.

[81]  Yi Pan,et al.  Classification of Alzheimer's Disease Using Whole Brain Hierarchical Network , 2018, IEEE/ACM Transactions on Computational Biology and Bioinformatics.

[82]  Yi Pan,et al.  Classification of autism spectrum disorder by combining brain connectivity and deep neural network classifier , 2019, Neurocomputing.

[83]  Yi Pan,et al.  Deep convolutional neural network for automatically segmenting acute ischemic stroke lesion in multi-modality MRI , 2019, Neural Computing and Applications.

[84]  Hariharan Ravishankar,et al.  Joint Deep Learning of Foreground, Background and Shape for Robust Contextual Segmentation , 2017, IPMI.

[85]  Hao Chen,et al.  3D multi‐scale FCN with random modality voxel dropout learning for Intervertebral Disc Localization and Segmentation from Multi‐modality MR Images , 2018, Medical Image Anal..

[86]  D. Louis Collins,et al.  Review of automatic segmentation methods of multiple sclerosis white matter lesions on conventional magnetic resonance imaging , 2013, Medical Image Anal..

[87]  D. Opitz,et al.  Popular Ensemble Methods: An Empirical Study , 1999, J. Artif. Intell. Res..

[88]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[89]  Shuiwang Ji,et al.  Residual Deconvolutional Networks for Brain Electron Microscopy Image Segmentation , 2017, IEEE Transactions on Medical Imaging.

[90]  Hao Chen,et al.  VoxResNet: Deep voxelwise residual networks for brain segmentation from 3D MR images , 2017, NeuroImage.

[91]  Thomas Brox,et al.  U-Net: deep learning for cell counting, detection, and morphometry , 2018, Nature Methods.

[92]  Yoshua Bengio,et al.  Attention-Based Models for Speech Recognition , 2015, NIPS.

[93]  Nick C Fox,et al.  Neuroimaging standards for research into small vessel disease and its contribution to ageing and neurodegeneration , 2013, The Lancet Neurology.

[94]  Sebastian Thrun,et al.  Dermatologist-level classification of skin cancer with deep neural networks , 2017, Nature.

[95]  Kim Mouridsen,et al.  Interrater Agreement for Final Infarct MRI Lesion Delineation , 2009, Stroke.

[96]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[97]  Manabu Inoue,et al.  MRI profile and response to endovascular reperfusion after stroke (DEFUSE 2): a prospective cohort study , 2012, The Lancet Neurology.

[98]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[99]  Sachin Mehta,et al.  3D-ESPNet with Pyramidal Refinement for Volumetric Brain Tumor Image Segmentation , 2018, BrainLes@MICCAI.

[100]  Ganapathy Krishnamurthi,et al.  Densely Connected Fully Convolutional Network for Short-Axis Cardiac Cine MR Image Segmentation and Heart Diagnosis Using Random Forest , 2017, STACOM@MICCAI.

[101]  Christos Davatzikos,et al.  Computer-assisted Segmentation of White Matter Lesions in 3d Mr Images Using Support Vector Machine 1 , 2022 .

[102]  Verónica Vilaplana,et al.  Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries , 2017, Lecture Notes in Computer Science.

[103]  Yaohang Li,et al.  Clinical big data and deep learning: Applications, challenges, and future outlooks , 2019, Big Data Min. Anal..

[104]  Linda G. Shapiro,et al.  ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation , 2018, ECCV.

[105]  Andreas K. Maier,et al.  Dilated Convolutions in Neural Networks for Left Atrial Segmentation in 3D Gadolinium Enhanced-MRI , 2018, STACOM@MICCAI.

[106]  Scott Hamilton,et al.  Magnetic resonance imaging profiles predict clinical response to early reperfusion: The diffusion and perfusion imaging evaluation for understanding stroke evolution (DEFUSE) study , 2006, Annals of neurology.

[107]  Lin Yang,et al.  MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[108]  Yasuo Yamashita,et al.  Computer-aided evaluation method of white matter hyperintensities related to subcortical vascular dementia based on magnetic resonance imaging , 2010, Comput. Medical Imaging Graph..

[109]  Christopher Joseph Pal,et al.  Learning normalized inputs for iterative estimation in medical image segmentation , 2017, Medical Image Anal..

[110]  Chi-Wing Fu,et al.  H-DenseUNet: Hybrid Densely Connected UNet for Liver and Tumor Segmentation From CT Volumes , 2018, IEEE Transactions on Medical Imaging.

[111]  Marc Pollefeys,et al.  An Exploration of 2D and 3D Deep Learning Techniques for Cardiac MR Image Segmentation , 2017, STACOM@MICCAI.

[112]  Liang Chen,et al.  DRINet for Medical Image Segmentation , 2018, IEEE Transactions on Medical Imaging.

[113]  Hao Chen,et al.  DCAN: Deep Contour-Aware Networks for Accurate Gland Segmentation , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[114]  Geena Kim,et al.  Brain Tumor Segmentation Using Deep Fully Convolutional Neural Networks , 2017, BrainLes@MICCAI.

[115]  Pham The Bao,et al.  Brain Tumor Segmentation Using Bit-plane and UNET , 2018, BrainLes@MICCAI.

[116]  Pascal Fua,et al.  A domain-adaptive two-stream U-Net for electron microscopy image segmentation , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[117]  Hao Chen,et al.  DCAN: Deep contour‐aware networks for object instance segmentation from histology images , 2017, Medical Image Anal..

[118]  Matthew T. Freedman,et al.  Artificial convolution neural network techniques and applications for lung nodule detection , 1995, IEEE Trans. Medical Imaging.

[119]  Peter F. Neher,et al.  TractSeg - Fast and accurate white matter tract segmentation , 2018, NeuroImage.

[120]  Yi Pan,et al.  Improving Alzheimer's Disease Classification by Combining Multiple Measures , 2018, IEEE/ACM Transactions on Computational Biology and Bioinformatics.