Longitudinal multiple sclerosis lesion segmentation: Resource and challenge

Abstract In conjunction with the ISBI 2015 conference, we organized a longitudinal lesion segmentation challenge providing training and test data to registered participants. The training data consisted of five subjects with a mean of 4.4 time‐points, and test data of fourteen subjects with a mean of 4.4 time‐points. All 82 data sets had the white matter lesions associated with multiple sclerosis delineated by two human expert raters. Eleven teams submitted results using state‐of‐the‐art lesion segmentation algorithms to the challenge, with ten teams presenting their results at the conference. We present a quantitative evaluation comparing the consistency of the two raters as well as exploring the performance of the eleven submitted results in addition to three other lesion segmentation algorithms. The challenge presented three unique opportunities: (1) the sharing of a rich data set; (2) collaboration and comparison of the various avenues of research being pursued in the community; and (3) a review and refinement of the evaluation metrics currently in use. We report on the performance of the challenge participants, as well as the construction and evaluation of a consensus delineation. The image data and manual delineations will continue to be available for download, through an evaluation website2 as a resource for future researchers in the area. This data resource provides a platform to compare existing methods in a fair and consistent manner to each other and multiple manual raters. HighlightsPublic lesion data base of 21 training data sets and 61 testing data sets.Fully automated evaluation website.Comparison between 14 state‐of‐the‐art algorithms and 2 manual delineators.

[1]  Ahmet Ekin.  Feature-based brain mid-sagittal plane detection by RANSAC , 2006, 2006 14th European Signal Processing Conference.

[2]  Brian B. Avants,et al.  N4ITK: Improved N3 Bias Correction , 2010, IEEE Transactions on Medical Imaging.

[3]  Alan C. Evans,et al.  MRI Simulation Based Evaluation and Classifications Methods , 1999, IEEE Trans. Medical Imaging.

[4]  Snehashis Roy,et al.  Random forest regression for magnetic resonance image synthesis , 2017, Medical Image Anal..

[5]  Rohit Bakshi,et al.  Magnetic Resonance Imaging Advances in Multiple Sclerosis , 2005, Journal of neuroimaging : official journal of the American Society of Neuroimaging.

[6]  Christian Confavreux,et al.  The clinical epidemiology of multiple sclerosis. , 2008, Neuroimaging clinics of North America.

[7]  Bernhard Hemmer,et al.  An automated tool for detection of FLAIR-hyperintense white-matter lesions in Multiple Sclerosis , 2012, NeuroImage.

[8]  D. Louis Collins,et al.  Temporally Consistent Probabilistic Detection of New Multiple Sclerosis Lesions in Brain MRI , 2013, IEEE Transactions on Medical Imaging.

[9]  Jennifer L. Cuzzocreo,et al.  Reconstruction of the human cerebral cortex robust to white matter lesions: Method and validation , 2014, Human brain mapping.

[10]  D. Louis Collins,et al.  Multiple Sclerosis lesion segmentation using an automated multimodal Graph Cut , 2016 .

[11]  M. Stella Atkins,et al.  Segmentation of multiple sclerosis lesions in intensity corrected multispectral MRI , 1996, IEEE Trans. Medical Imaging.

[12]  D. Louis Collins,et al.  Statistical Analysis of Longitudinal MRI Data: Applications for Detection of Disease Activity in MS , 2002, MICCAI.

[13]  T J Brady,et al.  1H Nuclear magnetic resonance imaging in multiple sclerosis. , 1983, Neurologic clinics.

[14]  Snehashis Roy,et al.  Longitudinal Patch-Based Segmentation of Multiple Sclerosis White Matter Lesions , 2015, MLMI.

[15]  Stephen M. Smith,et al.  Segmentation of brain MR images through a hidden Markov random field model and the expectation-maximization algorithm , 2001, IEEE Transactions on Medical Imaging.

[16]  Amir Alansary,et al.  MRBrainS Challenge: Online Evaluation Framework for Brain Image Segmentation in 3T MRI Scans , 2015, Comput. Intell. Neurosci..

[17]  Xiaogang Wang,et al.  Highly Efficient Forward and Backward Propagation of Convolutional Neural Networks for Pixelwise Classification , 2014, ArXiv.

[18]  Olivier Commowick,et al.  Longitudinal Intensity Normalization in Multiple Sclerosis Patients , 2014, CLIP@MICCAI.

[19]  Guillermo Sapiro,et al.  Supervised Dictionary Learning , 2008, NIPS.

[20]  Hervé Delingette,et al.  Automatic Detection and Segmentation of Evolving Processes in 3D Medical Images: Application to Multiple Sclerosis , 1999, IPMI.

[21]  Christophe Collet,et al.  Lesions detection on 3D brain MRI using trimmmed likelihood estimator and probabilistic atlas , 2008, 2008 5th IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[22]  Michael Brady,et al.  Towards Realtime Multimodal Fusion for Image-Guided Interventions Using Self-similarities , 2013, MICCAI.

[23]  Hans-Hellmut Nagel,et al.  New likelihood test methods for change detection in image sequences , 1984, Comput. Vis. Graph. Image Process..

[24]  Hayit Greenspan,et al.  Patch-Based Segmentation with Spatial Consistency: Application to MS Lesions in Brain MRI , 2016, Int. J. Biomed. Imaging.

[25]  Pierre Geurts,et al.  Extremely randomized trees , 2006, Machine Learning.

[26]  A. Oliver,et al.  A toolbox for multiple sclerosis lesion segmentation , 2015, Neuroradiology.

[27]  Lisa Tang,et al.  Deep 3D Convolutional Encoder Networks With Shortcuts for Multiscale Feature Integration Applied to Multiple Sclerosis Lesion Segmentation , 2016, IEEE Transactions on Medical Imaging.

[28]  Brian B. Avants,et al.  Symmetric diffeomorphic image registration with cross-correlation: Evaluating automated labeling of elderly and neurodegenerative brain , 2008, Medical Image Anal..

[29]  Xavier Lladó,et al.  Automated tissue segmentation of MR brain images in the presence of white matter lesions , 2017, Medical Image Anal..

[30]  Matthew J. McAuliffe,et al.  Medical Image Processing, Analysis and Visualization in clinical research , 2001, Proceedings 14th IEEE Symposium on Computer-Based Medical Systems. CBMS 2001.

[31]  Koen L. Vincken,et al.  Probabilistic segmentation of white matter lesions in MR imaging , 2004, NeuroImage.

[32]  Osamu Sakai,et al.  Radiological Evaluation of Cerebral Aneurysms in Selected Clinical Presentations , 2005, Journal of neuroimaging : official journal of the American Society of Neuroimaging.

[33]  C BezdekJames.  A Convergence Theorem for the Fuzzy ISODATA Clustering Algorithms , 1980 .

[34]  Simon K. Warfield,et al.  A new classifier feature space for an improved Multiple Sclerosis lesion segmentation , 2011, 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[35]  Devrim Ünay,et al.  Concordance between computer-based neuroimaging findings and expert assessments in dementia grading , 2013, 2013 21st Signal Processing and Communications Applications Conference (SIU).

[36]  Mohammad Ali Sahraian,et al.  MRI Atlas of MS Lesions , 2007 .

[37]  Chowdhury,et al.  Medical Image Computing and Computer-Assisted Intervention – MICCAI 2008 , 2008, Lecture Notes in Computer Science.

[38]  Pierre-Louis Bazin,et al.  Homeomorphic brain image segmentation with topological and statistical atlases , 2008, Medical Image Anal..

[39]  D. Paty,et al.  Magnetic Resonance Imaging in the Assessment of Disease Activity in Multiple Sclerosis , 1988, Canadian Journal of Neurological Sciences / Journal Canadien des Sciences Neurologiques.

[40]  Frederik Barkhof,et al.  Can MS lesion stages be distinguished with MRI? A postmortem MRI and histopathology study , 2015, Journal of Neurology.

[41]  F. Wilcoxon.  Individual Comparisons by Ranking Methods , 1945 .

[42]  Snehashis Roy,et al.  Subject-Specific Sparse Dictionary Learning for Atlas-Based Brain MRI Segmentation , 2015, IEEE Journal of Biomedical and Health Informatics.

[43]  Massimo Filippi,et al.  Association between pathological and MRI findings in multiple sclerosis , 2012, The Lancet Neurology.

[44]  F. Fazekas,et al.  Recommendations to improve imaging and analysis of brain lesion load and atrophy in longitudinal studies of multiple sclerosis , 2012, Journal of Neurology.

[45]  H. Handels,et al.  Extra Tree forests for sub-acute ischemic stroke lesion segmentation in MR sequences , 2015, Journal of Neuroscience Methods.

[46]  C. Poser,et al.  Diagnostic criteria for multiple sclerosis , 2001, Clinical Neurology and Neurosurgery.

[47]  V. Vasily.  Global, regional, and national age–sex specific all-cause and cause-specific mortality for 240 causes of death, 1990–2013: a systematic analysis for the Global Burden of Disease Study 2013 , 2015, The Lancet.

[48]  Alex Rovira,et al.  Segmentation of multiple sclerosis lesions in brain MRI: A review of automated approaches , 2012, Inf. Sci..

[49]  Bernadette A. Thomas,et al.  Global, regional, and national age–sex specific all-cause and cause-specific mortality for 240 causes of death, 1990–2013: a systematic analysis for the Global Burden of Disease Study 2013 , 2015, The Lancet.

[50]  James C. Bezdek,et al.  A Convergence Theorem for the Fuzzy ISODATA Clustering Algorithms , 1980, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[51]  GreenspanHayit,et al.  Patch-based segmentation with spatial consistency , 2016 .

[52]  Pierre-Louis Bazin,et al.  Free software tools for atlas-based volumetric neuroimage analysis , 2005, SPIE Medical Imaging.

[53]  R. He,et al.  Unified Approach for Multiple Sclerosis Lesion Segmentation on Brain MRI , 2006, Annals of Biomedical Engineering.

[54]  Grégoire Malandain,et al.  Hierarchical segmentation of multiple sclerosis lesions in multi-sequence MRI , 2004, 2004 2nd IEEE International Symposium on Biomedical Imaging: Nano to Macro (IEEE Cat No. 04EX821).

[55]  Johan Montagnat,et al.  Automated Estimation of Brain Volume in Multiple Sclerosis with BICCR , 2001, IPMI.

[56]  Simon K. Warfield,et al.  A Model of Population and Subject (MOPS) Intensities With Application to Multiple Sclerosis Lesion Segmentation , 2015, IEEE Transactions on Medical Imaging.

[57]  M. Horsfield,et al.  Quantitative assessment of MRI lesion load in monitoring the evolution of multiple sclerosis. , 1995, Brain : a journal of neurology.

[58]  Fabrice Heitz,et al.  Automatic change detection in multimodal serial MRI: application to multiple sclerosis lesion evolution , 2003, NeuroImage.

[59]  Raj Shekhar,et al.  Medical Image Processing , 2010, Handbook of Signal Processing Systems.

[60]  et al.,et al.  ISLES 2015 ‐ A public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI , 2017, Medical Image Anal..

[61]  D. Louis Collins,et al.  Trimmed-Likelihood Estimation for Focal Lesions and Tissue Segmentation in Multisequence MRI for Multiple Sclerosis , 2011, IEEE Transactions on Medical Imaging.

[62]  Ronald M. Summers,et al.  A New 2.5D Representation for Lymph Node Detection Using Random Sets of Deep Convolutional Neural Network Observations , 2014, MICCAI.

[63]  Guido Gerig,et al.  Spatio-temporal Segmentation of Active Multiple Sclerosis Lesions in Serial MRI Data , 2001, IPMI.

[64]  Massimo Filippi,et al.  MRI techniques to monitor MS evolution: The present and the future , 2002, Neurology.

[65]  Peiqing Qian,et al.  Heterogeneity in longitudinal evolution of ring‐enhancing multiple sclerosis lesions , 2011, Annals of neurology.

[66]  Adam Finkelstein,et al.  The Generalized PatchMatch Correspondence Algorithm , 2010, ECCV.

[67]  D. Louis Collins,et al.  Design and construction of a realistic digital brain phantom , 1998, IEEE Transactions on Medical Imaging.

[68]  Peter A. Calabresi,et al.  OASIS is Automated Statistical Inference for Segmentation, with applications to multiple sclerosis lesion segmentation in MRI☆ , 2013, NeuroImage: Clinical.

[69]  Peter A. Calabresi,et al.  A topology-preserving approach to the segmentation of brain images with multiple sclerosis lesions , 2010, NeuroImage.

[70]  Martin Styner,et al.  Standardized evaluation methodology and reference database for evaluating coronary artery centerline extraction algorithms , 2009, Medical Image Anal..

[71]  Massimo Filippi,et al.  Magnetic resonance outcome measures in multiple sclerosis trials: time to rethink? , 2014, Current opinion in neurology.

[72]  D. Louis Collins,et al.  Optimized PatchMatch for Near Real Time and Accurate Label Fusion , 2014, MICCAI.

[73]  Hayit Greenspan,et al.  Multiple Sclerosis Lesion Detection Using Constrained GMM and Curve Evolution , 2009, Int. J. Biomed. Imaging.

[74]  Christian Barillot,et al.  Adaptive dictionary learning for competitive classification of multiple sclerosis lesions , 2015, 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI).

[75]  C. Guttmann,et al.  MR Imaging Intensity Modeling of Damage and Repair In Multiple Sclerosis: Relationship of Short-Term Lesion Recovery to Progression and Disability , 2007, American Journal of Neuroradiology.

[76]  Sébastien Ourselin,et al.  Bayesian Model Selection for Pathological Neuroimaging Data Applied to White Matter Lesion Segmentation , 2015, IEEE Transactions on Medical Imaging.

[77]  R. Kikinis,et al.  Quantitative follow‐up of patients with multiple sclerosis using MRI: Technical aspects , 1999, Journal of magnetic resonance imaging : JMRI.

[78]  L. R. Dice.  Measures of the Amount of Ecologic Association Between Species , 1945 .

[79]  Marco Battaglini,et al.  Automated identification of brain new lesions in multiple sclerosis using subtraction images , 2014, Journal of magnetic resonance imaging : JMRI.

[80]  Xiaodong Tao,et al.  White matter lesion segmentation using machine learning and weakly labeled MR images , 2011, Medical Imaging.

[81]  Xavier Lladó,et al.  A subtraction pipeline for automatic detection of new appearing multiple sclerosis lesions in longitudinal studies , 2014, Neuroradiology.

[82]  K. Pearson.  VII. Note on regression and inheritance in the case of two parents , 1895, Proceedings of the Royal Society of London.

[83]  Aaron Carass,et al.  A JOINT REGISTRATION AND SEGMENTATION APPROACH TO SKULL STRIPPING , 2007, 2007 4th IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[84]  C M Crainiceanu,et al.  Automatic Lesion Incidence Estimation and Detection in Multiple Sclerosis Using Multisequence Longitudinal MRI , 2013, American Journal of Neuroradiology.

[85]  Snehashis Roy,et al.  Example based lesion segmentation , 2014, Medical Imaging.

[86]  Snehashis Roy,et al.  MR contrast synthesis for lesion segmentation , 2010, 2010 IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[87]  Brian B. Avants,et al.  An Open Source Multivariate Framework for n-Tissue Segmentation with Evaluation on Public Data , 2011, Neuroinformatics.

[88]  Martin Styner,et al.  Comparison and Evaluation of Methods for Liver Segmentation From CT Datasets , 2009, IEEE Transactions on Medical Imaging.

[89]  W. L. Benedict,et al.  Multiple Sclerosis , 2007, Journal - Michigan State Medical Society.

[90]  Daniel Rueckert,et al.  Multiple Sclerosis Lesion Segmentation Using Dictionary Learning and Sparse Coding , 2013, MICCAI.

[91]  Olivier Clatz,et al.  Spatial decision forests for MS lesion segmentation in multi-channel magnetic resonance images , 2011, NeuroImage.

[92]  Olivier Clatz,et al.  Spatial Decision Forests for MS Lesion Segmentation in Multi-Channel MR Images , 2010, MICCAI.

[93]  Saurabh Jain,et al.  Automatic segmentation and volumetry of multiple sclerosis brain lesions from MR images , 2015, NeuroImage: Clinical.

[94]  D. Louis Collins,et al.  Bayesian Classification of Multiple Sclerosis Lesions in Longitudinal MRI Using Subtraction Images , 2010, MICCAI.

[95]  Koenraad Van Leemput,et al.  Automated segmentation of multiple sclerosis lesions by model outlier detection , 2001, IEEE Transactions on Medical Imaging.

[96]  Brian B. Avants,et al.  The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) , 2015, IEEE Transactions on Medical Imaging.

[97]  Sébastien Ourselin,et al.  Global image registration using a symmetric block-matching approach , 2014, Journal of medical imaging.

[98]  Farzad Towhidkhah,et al.  Fully automatic segmentation of multiple sclerosis lesions in brain MR FLAIR images using adaptive mixtures method and markov random field model , 2008, Comput. Biol. Medicine.

[99]  B. Ginneken,et al.  3D Segmentation in the Clinic: A Grand Challenge , 2007 .

[100]  D. Louis Collins,et al.  Model-based 3-D segmentation of multiple sclerosis lesions in magnetic resonance brain images , 1995, IEEE Trans. Medical Imaging.

[101]  Doina Precup,et al.  IMaGe: Iterative Multilevel Probabilistic Graphical Model for Detection and Segmentation of Multiple Sclerosis Lesions in Brain MRI , 2015, IPMI.

[102]  Bruce Fischl,et al.  Avoiding asymmetry-induced bias in longitudinal image processing , 2011, NeuroImage.

[103]  Daniel S Reich,et al.  Evolution of the blood–brain barrier in newly forming multiple sclerosis lesions , 2011, Annals of neurology.

[104]  B.R. Sajja,et al.  A unified approach for lesion segmentation on MRI of multiple sclerosis , 2004, The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[105]  William M. Wells,et al.  Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation , 2004, IEEE Transactions on Medical Imaging.

[106]  John C. Mazziotta,et al.  A Probabilistic Atlas and Reference System for the Human Brain , 2001 .

[107]  Simon K. Warfield,et al.  Population intensity outliers or a new model for brain WM abnormalities , 2012, 2012 9th IEEE International Symposium on Biomedical Imaging (ISBI).

[108]  Pierre Geurts,et al.  Inferring biological networks with output kernel trees , 2007, BMC Bioinformatics.

[109]  D. Arnold,et al.  Combining Robust Expectation Maximization and Mean Shift algorithms for Multiple Sclerosis Brain Segmentation , 2008 .

[110]  Ying Wu,et al.  Automated segmentation of multiple sclerosis lesion subtypes with multichannel MRI , 2006, NeuroImage.

[111]  Amod Jog,et al.  Multi-output decision trees for lesion segmentation in multiple sclerosis , 2015, Medical Imaging.

[112]  A. C. Cem Say.  L'Hôpital's Filter for QSIM , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[113]  Ron Kikinis,et al.  Adaptive Template Moderated Spatially Varying Statistical Classification , 1998, MICCAI.

[114]  D. Louis Collins,et al.  A Generative Model for Automatic Detection of Resolving Multiple Sclerosis Lesions , 2014, BAMBI.

[115]  Benoit M. Dawant,et al.  Morphometric analysis of white matter lesions in MR images: method and validation , 1994, IEEE Trans. Medical Imaging.

[116]  Alan C. Evans,et al.  The Role of MRI in clinical trials of multiple sclerosis: Comparison of image processing techniques , 1997, Annals of neurology.

[117]  Janet Elizabeth Hope.  Open Source , 2017, Encyclopedia of GIS.

[118]  Jeffrey A. Cohen,et al.  Diagnostic criteria for multiple sclerosis: 2010 Revisions to the McDonald criteria , 2011, Annals of neurology.

[119]  A. Evans,et al.  MRI simulation-based evaluation of image-processing and classification methods , 1999, IEEE Transactions on Medical Imaging.

[120]  Snehashis Roy,et al.  Subject Specific Sparse Dictionary Learning for Atlas Based Brain MRI Segmentation , 2014, MLMI.

[121]  Alan C. Evans,et al.  BrainWeb: Online Interface to a 3D MRI Simulated Brain Database , 1997 .

[122]  Matthew D. Zeiler.  ADADELTA: An Adaptive Learning Rate Method , 2012, ArXiv.

[123]  Michael Brady,et al.  MIND: Modality independent neighbourhood descriptor for multi-modal deformable registration , 2012, Medical Image Anal..

[124]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[125]  Y Ge,et al.  Enhancing patterns in multiple sclerosis: evolution and persistence. , 2001, AJNR. American journal of neuroradiology.

[126]  Mohammad Havaei,et al.  HeMIS: Hetero-Modal Image Segmentation , 2016, MICCAI.

[127]  Dhanesh Ramachandram,et al.  Automatic white matter lesion segmentation using an adaptive outlier detection method. , 2012, Magnetic resonance imaging.

[128]  Aaron Carass,et al.  Erratum to: The Java Image Science Toolkit (JIST) for Rapid Prototyping and Publishing of Neuroimaging Software , 2010, Neuroinformatics.

[129]  Olivier Salvado,et al.  Lesion segmentation from multimodal MRI using random forest following ischemic stroke , 2014, NeuroImage.

[130]  J Mazziotta,et al.  A probabilistic atlas and reference system for the human brain: International Consortium for Brain Mapping (ICBM). , 2001, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[131]  D. Louis Collins,et al.  Bayesian MS Lesion Classification Modeling Regional and Local Spatial Information , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[132]  Aaron Carass,et al.  Simple paradigm for extra-cerebral tissue removal: Algorithm and analysis , 2011, NeuroImage.

[133]  Pierrick Coupé,et al.  An Optimized Blockwise Nonlocal Means Denoising Filter for 3-D Magnetic Resonance Images , 2008, IEEE Transactions on Medical Imaging.

[134]  Bassem A. Abdullah.  Segmentation of Multiple Sclerosis Lesions in Brain MRI , 2012 .

[135]  D. Louis Collins,et al.  Review of automatic segmentation methods of multiple sclerosis white matter lesions on conventional magnetic resonance imaging , 2013, Medical Image Anal..

[136]  Supun Samarasekera,et al.  Multiple sclerosis lesion quantification using fuzzy-connectedness principles , 1997, IEEE Transactions on Medical Imaging.

[137]  Lisa Tang,et al.  Deep Convolutional Encoder Networks for Multiple Sclerosis Lesion Segmentation , 2015, MICCAI.

[138]  Pierre Hellier,et al.  STREM: A Robust Multidimensional Parametric Method to Segment MS Lesions in MRI , 2005, MICCAI.

[139]  Dinggang Shen,et al.  CLASSIC: Consistent Longitudinal Alignment and Segmentation for Serial Image Computing , 2005, IPMI.

[140]  Joseph Ross Mitchell,et al.  Segmentation of multiple sclerosis lesions using support vector machines , 2003, SPIE Medical Imaging.