Task-based evaluation of segmentation algorithms for diffusion-weighted MRI without using a gold standard

In many studies, the estimation of the apparent diffusion coefficient (ADC) of lesions in visceral organs in diffusion-weighted (DW) magnetic resonance images requires an accurate lesion-segmentation algorithm. To evaluate these lesion-segmentation algorithms, region-overlap measures are used currently. However, the end task from the DW images is accurate ADC estimation, and the region-overlap measures do not evaluate the segmentation algorithms on this task. Moreover, these measures rely on the existence of gold-standard segmentation of the lesion, which is typically unavailable. In this paper, we study the problem of task-based evaluation of segmentation algorithms in DW imaging in the absence of a gold standard. We first show that using manual segmentations instead of gold-standard segmentations for this task-based evaluation is unreliable. We then propose a method to compare the segmentation algorithms that does not require gold-standard or manual segmentation results. The no-gold-standard method estimates the bias and the variance of the error between the true ADC values and the ADC values estimated using the automated segmentation algorithm. The method can be used to rank the segmentation algorithms on the basis of both the ensemble mean square error and precision. We also propose consistency checks for this evaluation technique.

[1]  P. Jaccard.  THE DISTRIBUTION OF THE FLORA IN THE ALPINE ZONE.1 , 1912 .

[2]  L. R. Dice.  Measures of the Amount of Ecologic Association Between Species , 1945 .

[3]  G. J. Hahn,et al.  Statistical models in engineering , 1967 .

[4]  R. Gnanadesikan,et al.  Probability plotting methods for the analysis of data. , 1968, Biometrika.

[5]  Nuggehally Sampath Jayant,et al.  An adaptive clustering algorithm for image segmentation , 1989, International Conference on Acoustics, Speech, and Signal Processing,.

[6]  M. Bronskill,et al.  Receiver Operator characteristic (ROC) Analysis without Truth , 1990, Medical decision making : an international journal of the Society for Medical Decision Making.

[7]  H H Barrett,et al.  Objective assessment of image quality: effects of quantum noise and object variability. , 1990, Journal of the Optical Society of America. A, Optics and image science.

[8]  T. Coleman,et al.  On the Convergence of Reflective Newton Methods for Large-scale Nonlinear Minimization Subject to Bounds , 1992 .

[9]  Thomas F. Coleman,et al.  On the convergence of interior-reflective Newton methods for nonlinear minimization subject to bounds , 1994, Math. Program..

[10]  M. Horsfield,et al.  Intra- and inter-observer agreement of brain MRI lesion volume measurements in multiple sclerosis. A comparison of techniques. , 1995, Brain : a journal of neurology.

[11]  N. L. Johnson,et al.  Continuous Univariate Distributions. , 1995 .

[12]  David B. Dunson,et al.  Bayesian Data Analysis , 2010 .

[13]  H. Gudbjartsson,et al.  The rician distribution of noisy mri data , 1995, Magnetic resonance in medicine.

[14]  R. F. Wagner,et al.  Objective assessment of image quality. II. Fisher information, Fourier crosstalk, and figures of merit for task performance. , 1995, Journal of the Optical Society of America. A, Optics, image science, and vision.

[15]  Yongmin Kim,et al.  A methodology for evaluation of boundary detection algorithms on medical images , 1997, IEEE Transactions on Medical Imaging.

[16]  Michael Ljungberg,et al.  Deformable Fourier surfaces for volume segmentation in SPECT , 1998, Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No.98EX170).

[17]  Maryellen L. Giger,et al.  Automated seeded lesion segmentation on digital mammograms , 1998, IEEE Transactions on Medical Imaging.

[18]  R. Edelman,et al.  A Phantom for diffusion‐weighted imaging of acute stroke , 1998, Journal of magnetic resonance imaging : JMRI.

[19]  H. Barrett,et al.  Objective assessment of image quality. III. ROC metrics, ideal observers, and likelihood-generating functions. , 1998, Journal of the Optical Society of America. A, Optics, image science, and vision.

[20]  R. Gillies,et al.  Early increases in breast tumor xenograft water mobility in response to paclitaxel therapy detected by non-invasive diffusion magnetic resonance imaging. , 1999, Neoplasia.

[21]  J. M. Taylor,et al.  Diffusion magnetic resonance imaging: an early surrogate marker of therapeutic efficacy in brain tumors. , 2000, Journal of the National Cancer Institute.

[22]  Stephen M. Smith,et al.  Segmentation of brain MR images through a hidden Markov random field model and the expectation-maximization algorithm , 2001, IEEE Transactions on Medical Imaging.

[23]  B. Frieden.  Probability, Statistical Optics and Data Testing: A Problem Solving Approach , 2001 .

[24]  Matthew A. Kupinski,et al.  Objective Comparison of Quantitative Imaging Modalities Without the Use of a Gold Standard , 2001, IPMI.

[25]  C. Bortner,et al.  Apoptotic volume decrease and the incredible shrinking cell , 2002, Cell Death and Differentiation.

[26]  H. Barrett,et al.  Objective comparison of quantitative imaging modalities without the use of a gold standard , 2002, IEEE Transactions on Medical Imaging.

[27]  Eric Clarkson,et al.  Estimation in medical imaging without a gold standard. , 2002, Academic radiology.

[28]  R. Bammer.  Basic principles of diffusion-weighted imaging. , 2003, European journal of radiology.

[29]  Matthew A. Kupinski,et al.  Evaluating estimation techniques in medical imaging without a gold standard: experimental validation , 2003, SPIE Medical Imaging.

[30]  Jeffrey A. Fessler,et al.  Fast, iterative image reconstruction for MRI in the presence of field inhomogeneities , 2003, IEEE Transactions on Medical Imaging.

[31]  Jie Tian,et al.  Automatic segmentation of brain infarction in diffusion-weighted MR images , 2003, SPIE Medical Imaging.

[32]  T. Huisman.  Diffusion-weighted imaging: basic concepts and application in cerebral stroke and head trauma , 2003, European Radiology.

[33]  J.J. Rodriguez,et al.  Snake-based liver lesion segmentation , 2004, 6th IEEE Southwest Symposium on Image Analysis and Interpretation, 2004..

[34]  David R. Anderson,et al.  Multimodel Inference , 2004 .

[35]  R. Gillies,et al.  Changes in water mobility measured by diffusion MRI predict response of metastatic breast cancer to chemotherapy. , 2004, Neoplasia.

[36]  William M. Wells,et al.  Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation , 2004, IEEE Transactions on Medical Imaging.

[37]  K. Zou,et al.  Three validation metrics for automated probabilistic image segmentation of brain tumours , 2004, Statistics in medicine.

[38]  Julian Z. Wang.  Inference : a note on estimation in the four-parameter beta distribution , 2005 .

[39]  A. Fenster,et al.  Evaluation of Segmentation algorithms for Medical Imaging , 2005, 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference.

[40]  Andreas Hadjiprocopis,et al.  Unbiased segmentation of diffusion-weighted magnetic resonance images of the brain using iterative clustering. , 2005, Magnetic resonance imaging.

[41]  Matthew A. Kupinski,et al.  Objective Assessment of Image Quality , 2005 .

[42]  Harrison H. Barrett,et al.  Foundations of Image Science , 2003, J. Electronic Imaging.

[43]  C. Thng,et al.  Liver Tumor Volume Estimation By Semi-Automatic Segmentation Method , 2005, 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference.

[44]  Eric Clarkson,et al.  Comparing cardiac ejection fraction estimation algorithms without a gold standard. , 2006, Academic radiology.

[45]  Jayaram K. Udupa,et al.  A framework for evaluating image segmentation algorithms , 2006, Comput. Medical Imaging Graph..

[46]  Oscar Camara,et al.  Generalized Overlap Measures for Evaluation and Validation in Medical Image Analysis , 2006, IEEE Transactions on Medical Imaging.

[47]  William M. Wells,et al.  Validation of image segmentation by estimating rater bias and variance , 2008, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[48]  Eric Clarkson,et al.  Estimating random signal parameters from noisy images with nuisance parameters: linear and scanning-linear methods. , 2008, Optics express.

[49]  Noureddine Doghmane,et al.  Brain MRI Segmentation and Lesions Detection by EM Algorithm , 2008 .

[50]  U. Motosugi,et al.  High‐b‐value diffusion‐weighted MR imaging of hepatocellular lesions: Estimation of grade of malignancy of hepatocellular carcinoma , 2009, Journal of magnetic resonance imaging : JMRI.

[51]  Shelly Lotenberg,et al.  Evaluation of uterine cervix segmentations using ground truth from multiple experts , 2009, Comput. Medical Imaging Graph..

[52]  M. Orton,et al.  Robust estimation of the apparent diffusion coefficient (ADC) in heterogeneous solid tumors , 2009, Magnetic resonance in medicine.

[53]  Yoshitsugu Matsumoto,et al.  A new phantom using polyethylene glycol as an apparent diffusion coefficient standard for MR imaging. , 2009, International journal of oncology.

[54]  Jian Xu,et al.  Diffusion‐weighted imaging of the liver: Comparison of navigator triggered and breathhold acquisitions , 2009, Journal of magnetic resonance imaging : JMRI.

[55]  Arvid Lundervold,et al.  Evaluation of automated brain MR image segmentation and volumetry methods , 2009, Human brain mapping.

[56]  Yoshitsugu Matsumoto,et al.  In vitro experimental study of the relationship between the apparent diffusion coefficient and changes in cellularity and cell morphology. , 2009, Oncology reports.

[57]  Matthew A. Kupinski,et al.  ADC Estimation in Multi-scan DWMRI , 2010 .

[58]  Matthew A Kupinski,et al.  ADC estimation of lesions in diffusion-weighted MR images: A maximum-likelihood approach , 2010, 2010 IEEE Southwest Symposium on Image Analysis & Interpretation (SSIAI).

[59]  Abhinav K Jha,et al.  A clustering algorithm for liver lesion segmentation of diffusion-weighted MR images , 2010, 2010 IEEE Southwest Symposium on Image Analysis & Interpretation (SSIAI).

[60]  S. Muda,et al.  Automated segmentation of brain lesion based on diffusion-weighted MRI using a split and merge approach , 2010, 2010 IEEE EMBS Conference on Biomedical Engineering and Sciences (IECBES).

[61]  Matthew A Kupinski,et al.  Evaluating segmentation algorithms for diffusion-weighted MR images: a task-based approach , 2010, Medical Imaging.

[62]  Ganesh Sundaramoorthi,et al.  Tubular Surface Segmentation for Extracting Anatomical Structures From Medical Imagery , 2010, IEEE Transactions on Medical Imaging.

[63]  Edward J. Dudewicz,et al.  Handbook of Fitting Statistical Distributions with R , 2010 .

[64]  Alfredo A. Romero.  Statistical Adequacy and Reliability of Inference in Regression-like Models , 2010 .

[65]  A. Skalski,et al.  Heart segmentation in echo images , 2011 .

[66]  Caroline Petitjean,et al.  A review of segmentation methods in short axis cardiac MR images , 2011, Medical Image Anal..

[67]  Xiuwen Liu,et al.  FISH Finder: a high-throughput tool for analyzing FISH images , 2011, Bioinform..

[68]  Jeffrey J. Rodriguez,et al.  A maximum-likelihood approach for ADC estimation of lesions in visceral organs , 2012, 2012 IEEE Southwest Symposium on Image Analysis and Interpretation.

[69]  R. Buxton.  The physics of functional magnetic resonance imaging (fMRI) , 2013, Reports on progress in physics. Physical Society.