Efficient multi‐scale 3D CNN with fully connected CRF for accurate brain lesion segmentation

HIGHLIGHTSAn efficient 11‐layers deep, multi‐scale, 3D CNN architecture.A novel training strategy that significantly boosts performance.The first employment of a 3D fully connected CRF for post‐processing.State‐of‐the‐art performance on three challenging lesion segmentation tasks.New insights into the automatically learned intermediate representations. ABSTRACT We propose a dual pathway, 11‐layers deep, three‐dimensional Convolutional Neural Network for the challenging task of brain lesion segmentation. The devised architecture is the result of an in‐depth analysis of the limitations of current networks proposed for similar applications. To overcome the computational burden of processing 3D medical scans, we have devised an efficient and effective dense training scheme which joins the processing of adjacent image patches into one pass through the network while automatically adapting to the inherent class imbalance present in the data. Further, we analyze the development of deeper, thus more discriminative 3D CNNs. In order to incorporate both local and larger contextual information, we employ a dual pathway architecture that processes the input images at multiple scales simultaneously. For post‐processing of the network's soft segmentation, we use a 3D fully connected Conditional Random Field which effectively removes false positives. Our pipeline is extensively evaluated on three challenging tasks of lesion segmentation in multi‐channel MRI patient data with traumatic brain injuries, brain tumours, and ischemic stroke. We improve on the state‐of‐the‐art for all three applications, with top ranking performance on the public benchmarks BRATS 2015 and ISLES 2015. Our method is computationally efficient, which allows its adoption in a variety of research and clinical settings. The source code of our implementation is made publicly available.

[1]  J. Stockman,et al.  Effect of early versus delayed interferon beta-1b treatment on disability after a first clinical event suggestive of multiple sclerosis: a 3-year follow-up analysis of the BENEFIT study , 2009 .

[2]  Olivier Salvado,et al.  Lesion segmentation from multimodal MRI using random forest following ischemic stroke , 2014, NeuroImage.

[3]  Konstantinos Kamnitsas,et al.  Multi-scale 3D convolutional neural networks for lesion segmentation in brain MRI , 2015 .

[4]  Sébastien Ourselin,et al.  Template-Based Multimodal Joint Generative Model of Brain Data , 2015, IPMI.

[5]  Adam R Ferguson,et al.  Quantitative CT improves outcome prediction in acute traumatic brain injury. , 2012, Journal of neurotrauma.

[6]  Leeanne M. Carey,et al.  Biologically reliable white matter fiber tractography: issues and solutions , 2013 .

[7]  Paul M. Thompson,et al.  Robust Brain Extraction Across Datasets and Comparison With Publicly Available Methods , 2011, IEEE Transactions on Medical Imaging.

[8]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Robust Semantic Pixel-Wise Labelling , 2015, CVPR 2015.

[9]  Jyrki Lötjönen,et al.  Robust whole-brain segmentation: Application to traumatic brain injury , 2015, Medical Image Anal..

[10]  Seunghoon Hong,et al.  Learning Deconvolution Network for Semantic Segmentation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[11]  Geoffrey E. Hinton,et al.  On the importance of initialization and momentum in deep learning , 2013, ICML.

[12]  Ben Glocker,et al.  Modality Propagation: Coherent Synthesis of Subject-Specific Scans with Data-Driven Regularization , 2013, MICCAI.

[13]  Yann LeCun,et al.  What is the best multi-stage architecture for object recognition? , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[14]  Brian B. Avants,et al.  The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) , 2015, IEEE Transactions on Medical Imaging.

[15]  Ronald M. Summers,et al.  DeepOrgan: Multi-level Deep Convolutional Networks for Automated Pancreas Segmentation , 2015, MICCAI.

[16]  Susan M. Chang,et al.  Updated response assessment criteria for high-grade gliomas: response assessment in neuro-oncology working group. , 2010, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[17]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[18]  J Whyte,et al.  Neuropsychological outcome and community re-integration following traumatic brain injury: The impact of frontal and non-frontal lesions , 2005, Brain injury.

[19]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[20]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[21]  Andrew Adams,et al.  Fast High‐Dimensional Filtering Using the Permutohedral Lattice , 2010, Comput. Graph. Forum.

[22]  Christian Igel,et al.  Deep Feature Learning for Knee Cartilage Segmentation Using a Triplanar Convolutional Neural Network , 2013, MICCAI.

[23]  Olivier Clatz,et al.  Spatial Decision Forests for MS Lesion Segmentation in Multi-Channel MR Images , 2010, MICCAI.

[24]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[25]  Robert Leech,et al.  Default mode network functional and structural connectivity after traumatic brain injury. , 2011, Brain : a journal of neurology.

[26]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[27]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[28]  Alex Rovira,et al.  MR in the diagnosis and monitoring of multiple sclerosis: an overview. , 2008, European journal of radiology.

[29]  Ronald M. Summers,et al.  A New 2.5D Representation for Lymph Node Detection Using Random Sets of Deep Convolutional Neural Network Observations , 2014, MICCAI.

[30]  Wiro J. Niessen,et al.  Brain tissue volumes in relation to cognitive function and risk of dementia , 2010, Neurobiology of Aging.

[31]  Daniel Rueckert,et al.  Multiple Sclerosis Lesion Segmentation Using Dictionary Learning and Sparse Coding , 2013, MICCAI.

[32]  Ludwig Kappos,et al.  Effect of early versus delayed interferon beta-1b treatment on disability after a first clinical event suggestive of multiple sclerosis: a 3-year follow-up analysis of the BENEFIT study , 2007, The Lancet.

[33]  Thomas Brox,et al.  Striving for Simplicity: The All Convolutional Net , 2014, ICLR.

[34]  Dazhe Zhao,et al.  Segmentation of Stroke Lesions in Multi-spectral MR Images Using Bias Correction Embedded FCM and Three Phase Level Set , 2015 .

[35]  Jeffrey S. Spence,et al.  Assessing spatial relationships between axonal integrity, regional brain volumes, and neuropsychological outcomes after traumatic axonal injury. , 2010, Journal of neurotrauma.

[36]  Yann LeCun,et al.  The Loss Surfaces of Multilayer Networks , 2014, AISTATS.

[37]  Vibhav Vineet,et al.  Conditional Random Fields as Recurrent Neural Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[38]  et al.,et al.  ISLES 2015 ‐ A public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI , 2017, Medical Image Anal..

[39]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[40]  Fiona Lecky,et al.  Collaborative European NeuroTrauma Effectiveness Research in Traumatic Brain Injury (CENTER-TBI): a prospective longitudinal observational study. , 2015, Neurosurgery.

[41]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[42]  Luca Maria Gambardella,et al.  Deep Neural Networks Segment Neuronal Membranes in Electron Microscopy Images , 2012, NIPS.

[43]  Y. Ge Multiple sclerosis: the role of MR imaging. , 2006, AJNR. American journal of neuroradiology.

[44]  Dinggang Shen,et al.  Deep Learning Based Imaging Data Completion for Improved Brain Disease Diagnosis , 2014, MICCAI.

[45]  Xiang Zhang,et al.  OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks , 2013, ICLR.

[46]  Dazhe Zhao,et al.  Segmentation of Ischemic Stroke Lesions in Multi-spectral MR Images Using Weighting Suppressed FCM and Three Phase Level Set , 2015, Brainles@MICCAI.

[47]  Christos Davatzikos,et al.  Joint Segmentation and Deformable Registration of Brain Scans Guided by a Tumor Growth Model , 2011, MICCAI.

[48]  Shuiwang Ji,et al.  Deep convolutional neural networks for multi-modality isointense infant brain image segmentation , 2015, NeuroImage.

[49]  Ben Glocker,et al.  Decision Forests for Tissue-Specific Segmentation of High-Grade Gliomas in Multi-channel MR , 2012, MICCAI.

[50]  Frederik Barkhof,et al.  Longitudinal Cognitive Decline in Subcortical Ischemic Vascular Disease – The LADIS Study , 2009, Cerebrovascular Diseases.

[51]  Lisa Tang,et al.  Deep Convolutional Encoder Networks for Multiple Sclerosis Lesion Segmentation , 2015, MICCAI.

[52]  Iasonas Kokkinos,et al.  Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs , 2014, ICLR.

[53]  Nitish Srivastava,et al.  Improving neural networks by preventing co-adaptation of feature detectors , 2012, ArXiv.

[54]  Colin Studholme,et al.  An overlap invariant entropy measure of 3D medical image alignment , 1999, Pattern Recognit..

[55]  Guido Gerig,et al.  Neuroimaging of structural pathology and connectomics in traumatic brain injury: Toward personalized outcome prediction☆ , 2012, NeuroImage: Clinical.

[56]  Giovanni Montana,et al.  Deep neural networks for anatomical brain segmentation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[57]  Fred A. Hamprecht,et al.  Multi-modal Brain Tumor Segmentation using Deep Convolutional Neural Networks , 2014 .

[58]  R. Mccoll,et al.  Cerebral atrophy after traumatic white matter injury: correlation with acute neuroimaging and outcome. , 2008, Journal of neurotrauma.

[59]  Daniel Rueckert,et al.  Contusion segmentation from subjects with Traumatic Brain Injury: A random forest framework , 2014, 2014 IEEE 11th International Symposium on Biomedical Imaging (ISBI).

[60]  Pavel Kisilev,et al.  A Cross Saliency Approach to Asymmetry-Based Tumor Detection , 2015, MICCAI.

[61]  Mert R. Sabuncu,et al.  Segmentation of Cerebrovascular Pathologies in Stroke Patients with Spatial and Shape Priors , 2014, MICCAI.

[62]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[63]  Rasmus Larsen,et al.  An Ensemble of 2D Convolutional Neural Networks for Tumor Segmentation , 2015, SCIA.

[64]  Carlos A. Silva,et al.  Fully Automatic Brain Tumor Segmentation from Multiple MR Sequences using Hidden Markov Fields and Variational , 2013 .

[65]  Antonio Criminisi,et al.  Segmentation of Brain Tumor Tissues with Convolutional Neural Networks , 2014 .

[66]  Victor Alves,et al.  Deep Convolutional Neural Networks for the Segmentation of Gliomas in Multi-sequence MRI , 2015, Brainles@MICCAI.

[67]  Bernhard Hemmer,et al.  An automated tool for detection of FLAIR-hyperintense white-matter lesions in Multiple Sclerosis , 2012, NeuroImage.

[68]  Luca Maria Gambardella,et al.  Mitosis Detection in Breast Cancer Histology Images with Deep Neural Networks , 2013, MICCAI.

[69]  Christopher Joseph Pal,et al.  Brain tumor segmentation with Deep Neural Networks , 2015, Medical Image Anal..

[70]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[71]  Alan Connelly,et al.  Beyond the lesion: neuroimaging foundations for post-stroke recovery , 2013 .

[72]  Koenraad Van Leemput,et al.  Automated model-based tissue classification of MR images of the brain , 1999, IEEE Transactions on Medical Imaging.

[73]  Guido Gerig,et al.  A brain tumor segmentation framework based on outlier detection , 2004, Medical Image Anal..

[74]  Vladlen Koltun,et al.  Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials , 2011, NIPS.

[75]  D. Louis Collins,et al.  Symmetric Atlasing and Model Based Segmentation: An Application to the Hippocampus in Older Adults , 2006, MICCAI.

[76]  Hanna-Leena Halme,et al.  ISLES (SISS) Challenge 2015: Segmentation of Stroke Lesions Using Spatial Normalization, Random Forest Classification and Contextual Clustering , 2015, Brainles@MICCAI.

[77]  Nikos Paragios,et al.  Joint Tumor Segmentation and Dense Deformable Registration of Brain MR Images , 2012, MICCAI.

[78]  Razvan Pascanu,et al.  Theano: new features and speed improvements , 2012, ArXiv.

[79]  Stephen R. Aylward,et al.  Low-Rank to the Rescue - Atlas-Based Analyses in the Presence of Pathologies , 2014, MICCAI.

[80]  Anne Vik,et al.  A longitudinal MRI study of traumatic axonal injury in patients with moderate and severe traumatic brain injury , 2012, Journal of Neurology, Neurosurgery & Psychiatry.

[81]  Bilwaj Gaonkar,et al.  GLISTRboost: Combining Multimodal MRI Segmentation, Registration, and Biophysical Tumor Growth Modeling with Gradient Boosting Machines for Glioma Segmentation , 2015, Brainles@MICCAI.