Fast and Precise Hippocampus Segmentation Through Deep Convolutional Neural Network Ensembles and Transfer Learning

Automatic segmentation of the hippocampus from 3D magnetic resonance imaging mostly relied on multi-atlas registration methods. In this work, we exploit recent advances in deep learning to design and implement a fully automatic segmentation method, offering both superior accuracy and fast result. The proposed method is based on deep Convolutional Neural Networks (CNNs) and incorporates distinct segmentation and error correction steps. Segmentation masks are produced by an ensemble of three independent models, operating with orthogonal slices of the input volume, while erroneous labels are subsequently corrected by a combination of Replace and Refine networks. We explore different training approaches and demonstrate how, in CNN-based segmentation, multiple datasets can be effectively combined through transfer learning techniques, allowing for improved segmentation quality. The proposed method was evaluated using two different public datasets and compared favorably to existing methodologies. In the EADC-ADNI HarP dataset, the correspondence between the method’s output and the available ground truth manual tracings yielded a mean Dice value of 0.9015, while the required segmentation time for an entire MRI volume was 14.8 seconds. In the MICCAI dataset, the mean Dice value increased to 0.8835 through transfer learning from the larger EADC-ADNI HarP dataset.

[1]  Sabina Sonia Tangaro,et al.  Integrating longitudinal information in hippocampal volume measurements for the early detection of Alzheimer's disease , 2016, NeuroImage.

[2]  Chen Sun,et al.  Revisiting Unreasonable Effectiveness of Data in Deep Learning Era , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[3]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[4]  Thomas Brox,et al.  3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation , 2016, MICCAI.

[5]  Max A. Viergever,et al.  Label Fusion in Atlas-Based Segmentation Using a Selective and Iterative Method for Performance Level Estimation (SIMPLE) , 2010, IEEE Transactions on Medical Imaging.

[6]  Alon Wolf,et al.  An Adaptive Home-Use Robotic Rehabilitation System for the Upper Body , 2014, IEEE Journal of Translational Engineering in Health and Medicine.

[7]  Aabhas Majumdar,et al.  BrainSegNet: a convolutional neural network architecture for automated segmentation of human brain structures , 2017, Journal of medical imaging.

[8]  Nikos Komodakis,et al.  Detect, Replace, Refine: Deep Structured Prediction for Pixel Wise Labeling , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Yoshua Bengio,et al.  How transferable are features in deep neural networks? , 2014, NIPS.

[10]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[11]  Jose Dolz,et al.  3D fully convolutional networks for subcortical segmentation in MRI: A large-scale study , 2016, NeuroImage.

[12]  M. Mohammed Thaha,et al.  Brain Tumor Segmentation Using Convolutional Neural Networks in MRI Images , 2019, Journal of Medical Systems.

[13]  Paul A. Yushkevich,et al.  Multi-Atlas Segmentation with Joint Label Fusion , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[15]  Jitendra Malik,et al.  Human Pose Estimation with Iterative Error Feedback , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[17]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[18]  Konstantinos Kamnitsas,et al.  Efficient multi‐scale 3D CNN with fully connected CRF for accurate brain lesion segmentation , 2016, Medical Image Anal..

[19]  Iasonas Kokkinos,et al.  Sub-cortical brain structure segmentation using F-CNN'S , 2016, 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI).

[20]  Kyong Hwan Jin,et al.  Fast and robust segmentation of the striatum using deep convolutional neural networks , 2016, Journal of Neuroscience Methods.

[21]  Giovanni Montana,et al.  Deep neural networks for anatomical brain segmentation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[22]  A. Dale,et al.  NeurotechniqueWhole Brain Segmentation : Automated Labeling of Neuroanatomical Structures , 2002 .

[23]  Stephen M. Smith,et al.  A Bayesian model of shape and appearance for subcortical brain segmentation , 2011, NeuroImage.

[24]  Yan Wang,et al.  Patch-Based Label Fusion with Structured Discriminant Embedding for Hippocampus Segmentation , 2018, Neuroinformatics.

[25]  Carlos Platero,et al.  Combining a Patch-based Approach with a Non-rigid Registration-based Label Fusion Method for the Hippocampal Segmentation in Alzheimer’s Disease , 2017, Neuroinformatics.

[26]  Jitendra Malik,et al.  Iterative Instance Segmentation , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[28]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[29]  Paul A. Yushkevich,et al.  Multi-atlas segmentation with joint label fusion and corrective learning—an open source implementation , 2013, Front. Neuroinform..

[30]  Huchuan Lu,et al.  Saliency Detection with Recurrent Fully Convolutional Networks , 2016, ECCV.

[31]  W. Scoville,et al.  LOSS OF RECENT MEMORY AFTER BILATERAL HIPPOCAMPAL LESIONS , 1957, Journal of neurology, neurosurgery, and psychiatry.

[32]  Michaël Sdika,et al.  Combining atlas based segmentation and intensity classification with nearest neighbor transform and accuracy weighted vote , 2010, Medical Image Anal..

[33]  Hao Chen,et al.  VoxResNet: Deep Voxelwise Residual Networks for Volumetric Brain Segmentation , 2016, ArXiv.

[34]  Yong Fan,et al.  Metric Learning for Multi-atlas based Segmentation of Hippocampus , 2016, Neuroinformatics.

[35]  Seyed-Ahmad Ahmadi,et al.  Hough-CNN: Deep learning for segmentation of deep brain regions in MRI and ultrasound , 2016, Comput. Vis. Image Underst..

[36]  Christopher Joseph Pal,et al.  Brain tumor segmentation with Deep Neural Networks , 2015, Medical Image Anal..

[37]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[38]  S. Resnick,et al.  Measuring Size and Shape of the Hippocampus in MR Images Using a Deformable Shape Model , 2002, NeuroImage.

[39]  Daniel Rueckert,et al.  Segmentation of MR images via discriminative dictionary learning and sparse coding: Application to hippocampus labeling , 2013, NeuroImage.

[40]  Stefan Carlsson,et al.  CNN Features Off-the-Shelf: An Astounding Baseline for Recognition , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[41]  James S. Duncan,et al.  Neighbor-constrained segmentation with level set based 3-D deformable models , 2004, IEEE Transactions on Medical Imaging.

[42]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[43]  D. Louis Collins,et al.  An Optimized PatchMatch for multi-scale and multi-feature label fusion , 2016, NeuroImage.

[44]  Paul J. Harrison The hippocampus in schizophrenia: a review of the neuropathological evidence and its pathophysiological implications , 2004, Psychopharmacology.

[45]  Nicos Maglaveras,et al.  Accurate and Fully Automatic Hippocampus Segmentation Using Subject-Specific 3D Optimal Local Maps Into a Hybrid Active Contour Model , 2014, IEEE Journal of Translational Engineering in Health and Medicine.

[46]  Nicos Maglaveras,et al.  Gradient-Based Reliability Maps for ACM-Based Segmentation of Hippocampus , 2014, IEEE Transactions on Biomedical Engineering.

[47]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Clément Farabet,et al.  Torch7: A Matlab-like Environment for Machine Learning , 2011, NIPS 2011.

[49]  Brian B. Avants,et al.  A learning-based wrapper method to correct systematic errors in automatic image segmentation: Consistently improved performance in hippocampus, cortex and brain segmentation , 2011, NeuroImage.

[50]  A Tateo,et al.  Multiple RF classifier for the hippocampus segmentation: Method and validation on EADC-ADNI Harmonized Hippocampal Protocol. , 2015, Physica medica : PM : an international journal devoted to the applications of physics to medicine and biology : official journal of the Italian Association of Biomedical Physics.

[51]  Nick C Fox,et al.  Clinical and biomarker changes in dominantly inherited Alzheimer's disease. , 2012, The New England journal of medicine.

[52]  Max A. Viergever,et al.  Automatic Segmentation of MR Brain Images With a Convolutional Neural Network , 2016, IEEE Transactions on Medical Imaging.

[53]  Daniel Rueckert,et al.  Multi-atlas based segmentation of brain images: Atlas selection and its effect on accuracy , 2009, NeuroImage.

[54]  Klaus H. Maier-Hein,et al.  Deep MRI brain extraction: A 3D convolutional neural network for skull stripping , 2016, NeuroImage.

[55]  Norbert Schuff,et al.  Automated cross-sectional and longitudinal hippocampal volume measurement in mild cognitive impairment and Alzheimer's disease , 2010, NeuroImage.

[56]  Shuiwang Ji,et al.  Deep convolutional neural networks for multi-modality isointense infant brain image segmentation , 2015, NeuroImage.

[57]  Xavier Lladó,et al.  Automated sub‐cortical brain structure segmentation combining spatial and deep convolutional features , 2017, Medical Image Anal..

[58]  Christian Wachinger,et al.  DeepNAT: Deep convolutional neural network for segmenting neuroanatomy , 2017, NeuroImage.

[59]  M. Viergever,et al.  Automatic Segmentation of MR Brain Images With a Convolutional Neural Network. , 2016, IEEE transactions on medical imaging.

[60]  N. Schuff,et al.  Magnetic resonance imaging of the entorhinal cortex and hippocampus in mild cognitive impairment and Alzheimer's disease , 2001, Journal of neurology, neurosurgery, and psychiatry.

[61]  Lisa Tang,et al.  Deep 3D Convolutional Encoder Networks With Shortcuts for Multiscale Feature Integration Applied to Multiple Sclerosis Lesion Segmentation , 2016, IEEE Transactions on Medical Imaging.

[62]  Liana G. Apostolova,et al.  Comparison of AdaBoost and Support Vector Machines for Detecting Alzheimer's Disease Through Automated Hippocampal Segmentation , 2010, IEEE Transactions on Medical Imaging.

[63]  Alan C. Evans,et al.  A nonparametric method for automatic correction of intensity nonuniformity in MRI data , 1998, IEEE Transactions on Medical Imaging.

[64]  D. Arnold,et al.  Mesial temporal damage in temporal lobe epilepsy: a volumetric MRI study of the hippocampus, amygdala and parahippocampal region. , 2003, Brain : a journal of neurology.

[65]  Sabina Sonia Tangaro,et al.  Automated hippocampal segmentation in 3D MRI using random undersampling with boosting algorithm , 2015, Pattern Analysis and Applications.

[66]  Nick C Fox,et al.  The Alzheimer's disease neuroimaging initiative (ADNI): MRI methods , 2008, Journal of magnetic resonance imaging : JMRI.

[67]  D. Louis Collins,et al.  Nonlocal Patch-Based Label Fusion for Hippocampus Segmentation , 2010, MICCAI.

[68]  Tao Sun,et al.  Accurate and Consistent Hippocampus Segmentation Through Convolutional LSTM and View Ensemble , 2017, MLMI@MICCAI.

[69]  A. Dale,et al.  Whole Brain Segmentation Automated Labeling of Neuroanatomical Structures in the Human Brain , 2002, Neuron.

[70]  James T. Becker,et al.  Quantitative Neuroimaging Software for Clinical Assessment of Hippocampal Volumes on MR Imaging , 2015, Journal of Alzheimer's disease : JAD.

[71]  John Tran,et al.  cuDNN: Efficient Primitives for Deep Learning , 2014, ArXiv.

[72]  Stephen M Smith,et al.  Fast robust automated brain extraction , 2002, Human brain mapping.

[73]  Jyrki Lötjönen,et al.  Robust whole-brain segmentation: Application to traumatic brain injury , 2015, Medical Image Anal..

[74]  D. Louis Collins,et al.  Training labels for hippocampal segmentation based on the EADC-ADNI harmonized hippocampal protocol , 2015, Alzheimer's & Dementia.

[75]  D. Louis Collins,et al.  Towards accurate, automatic segmentation of the hippocampus and amygdala from MRI by augmenting ANIMAL with a template library and label fusion , 2010, NeuroImage.

[76]  John G. Csernansky,et al.  Open Access Series of Imaging Studies (OASIS): Cross-sectional MRI Data in Young, Middle Aged, Nondemented, and Demented Older Adults , 2007, Journal of Cognitive Neuroscience.

[77]  Keith A. Johnson,et al.  Steps to standardization and validation of hippocampal volumetry as a biomarker in clinical trials and diagnostic criterion for Alzheimer’s disease , 2011, Alzheimer's & Dementia.

[78]  L. Staib,et al.  Hippocampal volume reduction in major depression. , 2000, The American journal of psychiatry.