Statistical Comparison of ROC Curves from Multiple Readers

Receiver operating characteristic (ROC) analysis is the commonly accepted method for comparing diagnostic imaging systems. In general, ROC studies are designed in such a way that multiple readers read the same images and each image is presented by means of two different imaging systems. Statistical methods for the comparison of the ROC curves from one reader have been developed, but extension of these meth ods to multiple readers is not straightforward. A new method of analysis is presented for the comparison of ROC curves from multiple readers. This method includes a non parametric estimation of the variances and covariances between the various areas under the curves. The method described is more appropriate than the paired t test, because it also takes the case-sample variation into account. Key words: ROC curves; ROC analysis; nonparametric estimation; area under the curve; comparison of ROC curves; EM algorithm. (Med Decis Making 1996;16:143-152)

[1]  John A. Swets,et al.  Evaluation of diagnostic systems : methods from signal detection theory , 1982 .

[2]  E. DeLong,et al.  Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. , 1988, Biometrics.

[3]  C. Metz.  ROC Methodology in Radiologic Imaging , 1986, Investigative radiology.

[4]  W. Hall,et al.  Confidence Bands for Receiver Operating Characteristic Curves , 1993, Medical decision making : an international journal of the Society for Medical Decision Making.

[5]  C E Metz,et al.  Some practical issues of experimental design and data analysis in radiological ROC studies. , 1989, Investigative radiology.

[6]  C. Metz,et al.  A New Approach for Testing the Significance of Differences Between ROC Curves Measured from Correlated Data , 1984 .

[7]  F P Ottes,et al.  Diagnostic evaluation of a PACS subsystem using mammographs. An ROC analysis. , 1988, Medical informatics = Medecine et informatique.

[8]  J. Hanley,et al.  A method of comparing the areas under receiver operating characteristic curves derived from the same cases. , 1983, Radiology.

[9]  Franklin A. Graybill,et al.  Introduction to The theory , 1974 .

[10]  J. Hanley,et al.  The meaning and use of the area under a receiver operating characteristic (ROC) curve. , 1982, Radiology.

[11]  C. Metz.  Basic principles of ROC analysis. , 1978, Seminars in nuclear medicine.

[12]  K. Berbaum,et al.  Receiver operating characteristic rating analysis. Generalization to the population of readers and patients with the jackknife method. , 1992, Investigative radiology.

[13]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .