An approach to comparing accuracies of two FLAIR MR sequences in the detection of multiple sclerosis lesions in the brain in the absence of gold standard.

RATIONALE AND OBJECTIVES The purpose of this study was to present a new methodology to compare accuracies of two imaging fluid attenuated inversion recovery (FLAIR) magnetic resonance sequences in detection of multiple sclerosis (MS) lesions in the brain in the absence of ground truth, and to determine whether the two sequences, which differed only in echo time (TE), have the same accuracy. MATERIALS AND METHODS We acquired FLAIR images at TE(1) = 90 ms and TE(2) = 155 ms from 46 patients with MS (24-69 years old, mean 45.8, 15 males) and 11 healthy volunteers (23-54 years old, mean 37.1, 6 males). Seven experienced neuroradiologists segmented lesions manually on randomly presented corresponding TE(1) and TE(2) images. For every image pair, a "surrogate ground truth" for each TE was generated by applying probability thresholds, ranging from 0.3 to 0.5, to the weighted average of experts' segmentations. Jackknife alternative free-response receiver operating characteristic analysis was used to compare experts' performance on TE(1) and TE(2) images, using successively the TE(1)- and TE(2)-based ground truths. RESULTS Supratentorially, there were significant differences in relative accuracy between the two sequences, ranging from 8.4% to 12.1%. In addition, we found a higher ratio of false positives to true positives for the TE(2) sequence using the TE(2) ground truth, compared to the TE(1) equivalent. Infratentorially, differences in the relative accuracy did not reach statistical significance. CONCLUSION The presented methodology may be useful in assessing the value of new clinical imaging protocols or techniques in the context of replacing existing ones, when the absolute ground truth is not available, and in determining changes in disease progression in follow-up studies. Our results suggest that the sequence with shorter TE should be preferred because it generates relatively fewer false positives. The finding is consistent with results of previous computer simulation studies.

[1]  Heinrich Lanfermann,et al.  Detection of lesions in multiple sclerosis by 2D FLAIR and single-slab 3D FLAIR sequences at 3.0 T: initial results , 2006, European Radiology.

[2]  David H. Miller,et al.  A longitudinal study of abnormalities on MRI and disability from multiple sclerosis. , 2002, The New England journal of medicine.

[3]  William M. Wells,et al.  Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation , 2004, IEEE Transactions on Medical Imaging.

[4]  M Filippi,et al.  Comparison of MR pulse sequences in the detection of multiple sclerosis lesions. , 1997, AJNR. American journal of neuroradiology.

[5]  G. Comi,et al.  Effect of laquinimod on MRI-monitored disease activity in patients with relapsing-remitting multiple sclerosis: a multicentre, randomised, double-blind, placebo-controlled phase IIb study , 2008, The Lancet.

[6]  A J Thompson,et al.  Multiple sclerosis lesion detection in the brain: A comparison of fast fluid-attenuated inversion recovery and conventional T2-weighted dual spin echo , 1997, Neurology.

[7]  K. Berbaum,et al.  Receiver operating characteristic rating analysis. Generalization to the population of readers and patients with the jackknife method. , 1992, Investigative radiology.

[8]  D P Chakraborty,et al.  Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data. , 1989, Medical physics.

[9]  Effect of T1 relaxation time on lesion contrast enhancement in flair MR imaging: a study using computer-generated brain maps. , 2001, AJR. American journal of roentgenology.

[10]  S Ropele,et al.  Clinically benign multiple sclerosis despite large T2 lesion load: Can we explain this paradox? , 2008, Multiple sclerosis.

[11]  David Gur,et al.  Performance assessments of diagnostic systems under the FROC paradigm: experimental, analytical, and results interpretation issues. , 2008, Academic radiology.

[12]  E. Melhem,et al.  Artificial multiple sclerosis lesions on simulated FLAIR brain MR images: echo time and observer performance in detection. , 2006, Radiology.

[13]  MR fluid-attenuated inversion recovery imaging as routine brain T2-weighted imaging. , 1999, European journal of radiology.

[14]  Frederik Barkhof,et al.  Subtraction MR images in a multiple sclerosis multicenter clinical trial setting. , 2009, Radiology.

[15]  W. Bradley,et al.  MRI: The Basics , 1997 .

[16]  N A Obuchowski,et al.  Sample size tables for receiver operating characteristic studies. , 2000, AJR. American journal of roentgenology.

[17]  Nancy A Obuchowski,et al.  Determining sample size for ROC studies: what is reasonable for the expected difference in tests' ROC areas? , 2003, Academic radiology.

[18]  Dev P Chakraborty,et al.  Validation and statistical power comparison of methods for analyzing free-response observer performance studies. , 2008, Academic radiology.

[19]  H. Schild,et al.  Imaging of inflammatory lesions at 3.0 Tesla in patients with clinically isolated syndromes suggestive of multiple sclerosis: a comparison of fluid-attenuated inversion recovery with T2 turbo spin-echo , 2006, European Radiology.

[20]  A. Compston,et al.  Recommended diagnostic criteria for multiple sclerosis: Guidelines from the international panel on the diagnosis of multiple sclerosis , 2001, Annals of neurology.

[21]  Rohit Bakshi,et al.  MRI in multiple sclerosis: current status and future prospects , 2008, The Lancet Neurology.

[22]  R. F. Wagner,et al.  Assessment of medical imaging systems and computer aids: a tutorial review. , 2007, Academic radiology.

[23]  S. Reingold,et al.  Diagnostic criteria for multiple sclerosis: 2005 revisions to the “McDonald Criteria” , 2005, Annals of neurology.

[24]  D. Chakraborty,et al.  Free-response methodology: alternate analysis and a new observer-performance experiment. , 1990, Radiology.

[25]  Frederik Barkhof,et al.  Intracortical lesions in multiple sclerosis: improved detection with 3D double inversion-recovery MR imaging. , 2005, Radiology.

[26]  R. F. Wagner,et al.  Assessment methodologies and statistical issues for computer-aided diagnosis of lung nodules in computed tomography: contemporary research topics relevant to the lung image database consortium. , 2004, Academic radiology.

[27]  G. Barker,et al.  Variations in T1 and T2 relaxation times of normal appearing white matter and lesions in multiple sclerosis , 2000, Journal of the Neurological Sciences.

[28]  George C. Ebers,et al.  MRI as an outcome in multiple sclerosis clinical trials. , 2009 .

[29]  H. Kundel,et al.  Reliability of soft-copy versus hard-copy interpretation of emergency department radiographs: a prototype study. , 2001, AJR. American journal of roentgenology.

[30]  Dev P Chakraborty,et al.  Observer studies involving detection and localization: modeling, analysis, and validation. , 2004, Medical physics.

[31]  Ludwig Kappos,et al.  Efficacy and safety of oral fumarate in patients with relapsing-remitting multiple sclerosis: a multicentre, randomised, double-blind, placebo-controlled phase IIb study , 2008, The Lancet.

[32]  S. Reingold,et al.  The role of magnetic resonance techniques in understanding and managing multiple sclerosis. , 1998, Brain : a journal of neurology.

[33]  Ludwig Kappos,et al.  A randomized, placebo-controlled trial of natalizumab for relapsing multiple sclerosis. , 2006, The New England journal of medicine.

[34]  Dev P Chakraborty.  Counterpoint to "Performance assessment of diagnostic systems under the FROC paradigm" by Gur and Rockette. , 2009, Academic radiology.

[35]  E. Melhem,et al.  Accuracy for detection of simulated lesions: comparison of fluid-attenuated inversion-recovery, proton density--weighted, and T2-weighted synthetic brain MR imaging. , 2001, AJR. American journal of roentgenology.

[36]  J. Sterne,et al.  Accuracy of magnetic resonance imaging for the diagnosis of multiple sclerosis: systematic review , 2006, BMJ : British Medical Journal.

[37]  Elias R Melhem,et al.  Detection of simulated multiple sclerosis lesions on T2-weighted and FLAIR images of the brain: observer performance. , 2006, Radiology.

[38]  B. McNeil,et al.  Assessment of radiologic tests: control of bias and other design considerations. , 1988, Radiology.

[39]  Alan C. Evans,et al.  The Role of MRI in clinical trials of multiple sclerosis: Comparison of image processing techniques , 1997, Annals of neurology.