Deep Learning Based Brain Tumor Segmentation: A Survey

Brain tumor segmentation is a challenging problem in medical image analysis. The goal of brain tumor segmentation is to generate accurate delineation of brain tumor regions with correctly located masks. In recent years, deep learning methods have shown very promising performance in solving various computer vision problems, such as image classification, object detection and semantic segmentation. A number of deep learning based methods have been applied to brain tumor segmentation and achieved impressive system performance. Considering state-of-the-art technologies and their performance, the purpose of this paper is to provide a comprehensive survey of recently developed deep learning based brain tumor segmentation techniques. The established works included in this survey extensively cover technical aspects such as the strengths and weaknesses of different approaches, pre- and post-processing frameworks, datasets and evaluation metrics. Finally, we conclude this survey by discussing the potential development in future research work.

[1]  Ender Konukoglu,et al.  Unsupervised Detection of Lesions in Brain MRI using constrained adversarial auto-encoders , 2018, ArXiv.

[2]  Guang Yang,et al.  MRI Brain Tumor Segmentation and Patient Survival Prediction Using Random Forests and Fully Convolutional Networks , 2017, BrainLes@MICCAI.

[3]  Leonidas J. Guibas,et al.  Taskonomy: Disentangling Task Transfer Learning , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[4]  Xavier Lladó,et al.  Deep convolutional neural networks for brain image analysis on magnetic resonance imaging: a review , 2017, Artif. Intell. Medicine.

[5]  Antonio Criminisi,et al.  Segmentation of Brain Tumor Tissues with Convolutional Neural Networks , 2014 .

[6]  Kunio Doi,et al.  Computer-aided diagnosis in medical imaging: Historical review, current status and future potential , 2007, Comput. Medical Imaging Graph..

[7]  Leonhard Held,et al.  Gaussian Markov Random Fields: Theory and Applications , 2005 .

[8]  Hao Chen,et al.  Automatic cerebral microbleeds detection from MR images via Independent Subspace Analysis based hierarchical features , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[9]  Christopher Joseph Pal,et al.  The Importance of Skip Connections in Biomedical Image Segmentation , 2016, LABELS/DLMIA@MICCAI.

[10]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[11]  Tao Xu,et al.  SegAN: Adversarial Network with Multi-scale L1 Loss for Medical Image Segmentation , 2017, Neuroinformatics.

[12]  B. S. Manjunath,et al.  Brain Tumor Segmentation and Tractographic Feature Extraction from Structural MR Images for Overall Survival Prediction , 2018, BrainLes@MICCAI.

[13]  Daniel L. Rubin,et al.  Deep Learning for Brain MRI Segmentation: State of the Art and Future Directions , 2017, Journal of Digital Imaging.

[14]  Mert R. Sabuncu,et al.  Unsupervised Learning for Fast Probabilistic Diffeomorphic Registration , 2018, MICCAI.

[15]  Klaus H. Maier-Hein,et al.  Brain Tumor Segmentation and Radiomics Survival Prediction: Contribution to the BRATS 2017 Challenge , 2017, BrainLes@MICCAI.

[16]  Verónica Vilaplana,et al.  Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries , 2017, Lecture Notes in Computer Science.

[17]  Xavier Lladó,et al.  Survival prediction using ensemble tumor segmentation and transfer learning , 2018, ArXiv.

[18]  Hongtu Zhu,et al.  TPCNN: Two-Phase Patch-Based Convolutional Neural Network for Automatic Brain Tumor Segmentation and Survival Prediction , 2017, BrainLes@MICCAI.

[19]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[20]  Andre Esteva,et al.  A guide to deep learning in healthcare , 2019, Nature Medicine.

[21]  Jürgen Schmidhuber,et al.  Parallel Multi-Dimensional LSTM, With Application to Fast Biomedical Volumetric Image Segmentation , 2015, NIPS.

[22]  Fred A. Hamprecht,et al.  Multi-modal Brain Tumor Segmentation using Deep Convolutional Neural Networks , 2014 .

[23]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[24]  Konstantinos Kamnitsas,et al.  Unsupervised domain adaptation in brain lesion segmentation with adversarial networks , 2016, IPMI.

[25]  Paolo Fiorini,et al.  Medical Robotics and Computer-Integrated Surgery , 2008, 2008 32nd Annual IEEE International Computer Software and Applications Conference.

[26]  Hongliang Ren,et al.  Multi-modal PixelNet for Brain Tumor Segmentation , 2017, BrainLes@MICCAI.

[27]  Ganapathy Krishnamurthi,et al.  Automatic Segmentation and Overall Survival Prediction in Gliomas Using Fully Convolutional Neural Network and Texture Analysis , 2017, BrainLes@MICCAI.

[28]  Sébastien Ourselin,et al.  Automatic Brain Tumor Segmentation Using Cascaded Anisotropic Convolutional Neural Networks , 2017, BrainLes@MICCAI.

[29]  LinLin Shen,et al.  Deep Learning Based Multimodal Brain Tumor Diagnosis , 2017, BrainLes@MICCAI.

[30]  Nassir Navab,et al.  Deep Autoencoding Models for Unsupervised Anomaly Segmentation in Brain MR Images , 2018, BrainLes@MICCAI.

[31]  Jianwei Zhang,et al.  A Modified U-Net for Brain MR Image Segmentation , 2018, ICCCS.

[32]  Jia Liu,et al.  A Cascaded Deep Convolutional Neural Network for Joint Segmentation and Genotype Prediction of Brainstem Gliomas , 2018, IEEE Transactions on Biomedical Engineering.

[33]  Ben Glocker,et al.  Decision Forests for Tissue-Specific Segmentation of High-Grade Gliomas in Multi-channel MR , 2012, MICCAI.

[34]  Sanja Fidler,et al.  Annotating Object Instances with a Polygon-RNN , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  John G. Csernansky,et al.  Open Access Series of Imaging Studies: Longitudinal MRI Data in Nondemented and Demented Older Adults , 2010, Journal of Cognitive Neuroscience.

[36]  Iasonas Kokkinos,et al.  Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs , 2014, ICLR.

[37]  Stephen J. McKenna,et al.  Boundary-Aware Fully Convolutional Network for Brain Tumor Segmentation , 2017, MICCAI.

[38]  Giovanni Montana,et al.  Deep neural networks for anatomical brain segmentation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[39]  Hao Chen,et al.  Automatic Detection of Cerebral Microbleeds From MR Images via 3D Convolutional Neural Networks , 2016, IEEE Transactions on Medical Imaging.

[40]  et al.,et al.  ISLES 2015 ‐ A public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI , 2017, Medical Image Anal..

[41]  Max A. Viergever,et al.  Automatic Segmentation of MR Brain Images With a Convolutional Neural Network , 2016, IEEE Transactions on Medical Imaging.

[42]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[43]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[44]  S. Sk A Survey of MRI-Based Brain Tumor Segmentation Methods , 2014 .

[45]  Sanja Fidler,et al.  Efficient Interactive Annotation of Segmentation Datasets with Polygon-RNN++ , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[46]  Marcus Liwicki,et al.  Scene labeling with LSTM recurrent neural networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[47]  Konstantinos Kamnitsas,et al.  Ensembles of Multiple Models and Architectures for Robust Brain Tumour Segmentation , 2017, BrainLes@MICCAI.

[48]  Klaus H. Maier-Hein,et al.  Deep MRI brain extraction: A 3D convolutional neural network for skull stripping , 2016, NeuroImage.

[49]  Konstantinos Kamnitsas,et al.  Efficient multi‐scale 3D CNN with fully connected CRF for accurate brain lesion segmentation , 2016, Medical Image Anal..

[50]  Vibhav Vineet,et al.  Conditional Random Fields as Recurrent Neural Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[51]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[52]  Thomas Grosges,et al.  Multimodal Brain Tumor Segmentation Using 3D Convolutional Networks , 2017, BrainLes@MICCAI.

[53]  G. Reifenberger,et al.  The 2016 World Health Organization Classification of Tumors of the Central Nervous System: a summary , 2016, Acta Neuropathologica.

[54]  Peter D. Chang,et al.  Fully Convolutional Deep Residual Neural Networks for Brain Tumor Segmentation , 2016, BrainLes@MICCAI.

[55]  Ben Glocker,et al.  Deep Generative Models in the Real-World: An Open Challenge from Medical Imaging , 2018, ArXiv.

[56]  Liang Chen,et al.  Fully automatic acute ischemic lesion segmentation in DWI using convolutional neural networks , 2017, NeuroImage: Clinical.

[57]  Konstantinos Kamnitsas,et al.  Unsupervised Lesion Detection in Brain CT using Bayesian Convolutional Autoencoders , 2018 .

[58]  Marios Savvides,et al.  Deep Recurrent Level Set for Segmenting Brain Tumors , 2018, MICCAI.

[59]  April Khademi,et al.  Assessing the Impact of Color Normalization in Convolutional Neural Network-Based Nuclei Segmentation Frameworks , 2019, Front. Bioeng. Biotechnol..

[60]  Zoubin Ghahramani,et al.  Probabilistic machine learning and artificial intelligence , 2015, Nature.

[61]  Simon Andermatt,et al.  Automated Segmentation of Multiple Sclerosis Lesions Using Multi-dimensional Gated Recurrent Units , 2017, BrainLes@MICCAI.

[62]  Ting Liu,et al.  Recent advances in convolutional neural networks , 2015, Pattern Recognit..

[63]  Lisa Tang,et al.  Deep 3D Convolutional Encoder Networks With Shortcuts for Multiscale Feature Integration Applied to Multiple Sclerosis Lesion Segmentation , 2016, IEEE Transactions on Medical Imaging.

[64]  Arno Klein,et al.  Evaluation of 14 nonlinear deformation algorithms applied to human brain MRI registration , 2009, NeuroImage.

[65]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[66]  Mark D Cicero,et al.  Training and Validating a Deep Convolutional Neural Network for Computer-Aided Detection and Classification of Abnormalities on Frontal Chest Radiographs , 2017, Investigative radiology.

[67]  Jonathan Ventura,et al.  Dilated Convolutions for Brain Tumor Segmentation in MRI Scans , 2017, BrainLes@MICCAI.

[68]  Yao Sun,et al.  A Strategy of MR Brain Tissue Images' Suggestive Annotation Based on Modified U-Net , 2018, ArXiv.

[69]  Ju Liu,et al.  Hybrid Pyramid U-Net Model for Brain Tumor Segmentation , 2018, Intelligent Information Processing.

[70]  Nico Karssemeijer,et al.  Deep multi-scale location-aware 3D convolutional neural networks for automated detection of lacunes of presumed vascular origin , 2016, NeuroImage: Clinical.

[71]  Yifan Yu,et al.  CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison , 2019, AAAI.

[72]  Tal Arbel,et al.  Brain Tumor Segmentation Using a 3D FCN with Multi-scale Loss , 2017, BrainLes@MICCAI.

[73]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[74]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[75]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[76]  Joong-Ho Won,et al.  Ensemble of Deep Convolutional Neural Networks for Prognosis of Ischemic Stroke , 2016, BrainLes@MICCAI.

[77]  Dinggang Shen,et al.  Deep Ensemble Sparse Regression Network for Alzheimer's Disease Diagnosis , 2016, MLMI@MICCAI.

[78]  Bruce R. Rosen,et al.  Sequential 3D U-Nets for Biologically-Informed Brain Tumor Segmentation , 2017, ArXiv.

[79]  Trevor Darrell,et al.  Deep Domain Confusion: Maximizing for Domain Invariance , 2014, CVPR 2014.

[80]  Dinggang Shen,et al.  State-space model with deep learning for functional dynamics estimation in resting-state fMRI , 2016, NeuroImage.

[81]  Alex Rovira,et al.  Improving automated multiple sclerosis lesion segmentation with a cascaded 3D convolutional neural network approach , 2017, NeuroImage.

[82]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[83]  T. Munich,et al.  Offline Handwriting Recognition with Multidimensional Recurrent Neural Networks , 2008, NIPS.

[84]  Sara Sedlar,et al.  Brain Tumor Segmentation Using a Multi-path CNN Based Method , 2017, BrainLes@MICCAI.

[85]  Camille Couprie,et al.  Semantic Segmentation using Adversarial Networks , 2016, NIPS 2016.

[86]  Koenraad Van Leemput,et al.  Automated model-based tissue classification of MR images of the brain , 1999, IEEE Transactions on Medical Imaging.

[87]  Karl J. Friston,et al.  Spatial Normalization using Basis Functions , 2003 .

[88]  Víctor M. Pérez-García,et al.  Towards Uncertainty-Assisted Brain Tumor Segmentation and Survival Prediction , 2017, BrainLes@MICCAI.

[89]  J. Sato,et al.  Using deep belief network modelling to characterize differences in brain morphometry in schizophrenia , 2016, Scientific Reports.

[90]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[91]  Yan Hu,et al.  3D Deep Neural Network-Based Brain Tumor Segmentation Using Multimodality Magnetic Resonance Sequences , 2017, BrainLes@MICCAI.

[92]  Christos Davatzikos,et al.  Advancing The Cancer Genome Atlas glioma MRI collections with expert segmentation labels and radiomic features , 2017, Scientific Data.

[93]  Victor Alves,et al.  Brain Tumor Segmentation Using Convolutional Neural Networks in MRI Images , 2016, IEEE Transactions on Medical Imaging.

[94]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[95]  Guang Yang,et al.  Automatic Brain Tumor Detection and Segmentation Using U-Net Based Fully Convolutional Networks , 2017, MIUA.

[96]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[97]  Yong Fan,et al.  A deep learning model integrating FCNNs and CRFs for brain tumor segmentation , 2017, Medical Image Anal..

[98]  Jose Dolz,et al.  Dense Multi-path U-Net for Ischemic Stroke Lesion Segmentation in Multiple Image Modalities , 2018, BrainLes@MICCAI.

[99]  Ganapathy Krishnamurthi,et al.  Multi-modal Brain Tumor Segmentation Using Stacked Denoising Autoencoders , 2015, Brainles@MICCAI.

[100]  Christopher Joseph Pal,et al.  Brain tumor segmentation with Deep Neural Networks , 2015, Medical Image Anal..

[101]  Vladlen Koltun,et al.  Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials , 2011, NIPS.

[102]  Liu Jin,et al.  A survey of MRI-based brain tumor segmentation methods , 2014 .

[103]  Kebin Jia,et al.  Multiscale CNNs for Brain Tumor Segmentation and Diagnosis , 2016, Comput. Math. Methods Medicine.

[104]  Zhi-Hua Zhou,et al.  Ieee Transactions on Knowledge and Data Engineering 1 Training Cost-sensitive Neural Networks with Methods Addressing the Class Imbalance Problem , 2022 .

[105]  Christoph Meinel,et al.  Conditional Adversarial Network for Semantic Segmentation of Brain Tumor , 2017, ArXiv.

[106]  Nelly Gordillo,et al.  State of the art survey on MRI brain tumor segmentation. , 2013, Magnetic resonance imaging.

[107]  Hao Li,et al.  Visualizing the Loss Landscape of Neural Nets , 2017, NeurIPS.

[108]  Sim Heng Ong,et al.  Focus, Segment and Erase: An Efficient Network for Multi-label Brain Tumor Segmentation , 2018, ECCV.

[109]  Stephen M. Smith,et al.  A Bayesian model of shape and appearance for subcortical brain segmentation , 2011, NeuroImage.

[110]  Yoshua Bengio,et al.  Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , 2015, ICML.

[111]  Hayit Greenspan,et al.  Multi-view longitudinal CNN for multiple sclerosis lesion segmentation , 2017, Eng. Appl. Artif. Intell..

[112]  Brian B. Avants,et al.  The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) , 2015, IEEE Transactions on Medical Imaging.

[113]  M. P. van den Heuvel,et al.  Deep learning predictions of survival based on MRI in amyotrophic lateral sclerosis , 2016, NeuroImage: Clinical.

[114]  Patrice Y. Simard,et al.  Best practices for convolutional neural networks applied to visual document analysis , 2003, Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings..

[115]  Lisa Tang,et al.  Deep Learning of Brain Lesion Patterns for Predicting Future Disease Activity in Patients with Early Symptoms of Multiple Sclerosis , 2016, LABELS/DLMIA@MICCAI.

[116]  Richard McKinley,et al.  Pooling-Free Fully Convolutional Networks with Dense Skip Connections for Semantic Segmentation, with Application to Brain Tumor Segmentation , 2017, BrainLes@MICCAI.

[117]  N. Karssemeijer,et al.  Discriminating solitary cysts from soft tissue lesions in mammography using a pretrained deep convolutional neural network , 2017, Medical physics.

[118]  Matti Pietikäinen,et al.  Deep Learning for Generic Object Detection: A Survey , 2018, International Journal of Computer Vision.

[119]  Zhaolin Chen,et al.  Residual Encoder and Convolutional Decoder Neural Network for Glioma Segmentation , 2017, BrainLes@MICCAI.

[120]  B. Scheithauer,et al.  The 2007 WHO classification of tumours of the central nervous system , 2007, Acta Neuropathologica.

[121]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[122]  Robert W. Miller,et al.  Brain Tumor Segmentation in MRI Scans Using Deeply-Supervised Neural Networks , 2017, BrainLes@MICCAI.

[123]  Pablo Arbeláez,et al.  Brain Tumor Segmentation and Parsing on MRIs Using Multiresolution Neural Networks , 2017, BrainLes@MICCAI.

[124]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[125]  Jinhua Yu,et al.  Brain Tumor Segmentation Using an Adversarial Network , 2017, BrainLes@MICCAI.

[126]  Mazhar Shaikh,et al.  Brain Tumor Segmentation Using Dense Fully Convolutional Neural Network , 2017, BrainLes@MICCAI.

[127]  Russell H. Taylor,et al.  Medical robotics in computer-integrated surgery , 2003, IEEE Trans. Robotics Autom..

[128]  Seunghoon Hong,et al.  Learning Deconvolution Network for Semantic Segmentation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).