A New Approach for Testing the Significance of Differences Between ROC Curves Measured from Correlated Data

Receiver Operating Characteristic analysis is now generally recognized as the most appropriate methodology for evaluating the diagnostic performance of medical imaging procedures (1–7). ROC analysis has been used in the field of psychophysics for three decades, and its theory and experimental methodology have been developed in considerable detail (8–13). Perhaps surprisingly, the statistical properties of ROC measures had received relatively little attention until several years ago, when the limited size of practical data sets in medical applications indicated the need for careful study of this issue. Recent progress in the statistical analysis of ROC data includes the work of Metz and Kronman (14,15), who developed a bivariate test for the statistical significance of differences between ROC curves measured from independent data sets; the work of Hanley and McNeil, who studied the statistical properties of the area under an ROC curve and developed techniques to predict the number of cases required tc demonstrate the significance of differences between ROC “Area Indexes” measured from either independent (16) or correlated (17) data sets; and the work of Swets and Pickett (7), who identified three components of variation in ROC measures and outlined a general statistical protocol for testing the significance of differences in the Area Index.

[1]  J. Swets,et al.  A decision-making theory of visual detection. , 1954, Psychological review.

[2]  J SWETS,et al.  Decision processes in perception. , 1961, Psychological review.

[3]  M. Kendall,et al.  The advanced theory of statistics , 1945 .

[4]  J. Swets.  Signal detection and recognition by human observers : contemporary readings , 1964 .

[5]  Irene A. Stegun,et al.  Handbook of Mathematical Functions. , 1966 .

[6]  D. M. Green,et al.  Signal detection theory and psychophysics , 1966 .

[7]  D. Dorfman,et al.  Maximum-likelihood estimation of parameters of signal-detection theory and determination of confidence intervals—Rating-method data , 1969 .

[8]  Byron J. T. Morgan,et al.  Some aspects of ROC curve-fitting: Normal and logistic models , 1972 .

[9]  K. S. Kölbig,et al.  Errata: Milton Abramowitz and Irene A. Stegun, editors, Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables, National Bureau of Standards, Applied Mathematics Series, No. 55, U.S. Government Printing Office, Washington, D.C., 1994, and all known reprints , 1972 .

[10]  J. Swets.  The Relative Operating Characteristic in Psychology , 1973, Science.

[11]  L B Lusted,et al.  Radiographic applications of receiver operating characteristic (ROC) curves. , 1974, Radiology.

[12]  E. Keeler,et al.  Primer on certain elements of medical decision making. , 1975, The New England journal of medicine.

[13]  James P. Egan,et al.  Signal detection theory and ROC analysis , 1975 .

[14]  Charles E. Metz,et al.  Progress in evaluation of human observer visual detection performance using the ROC curve approach , 1976 .

[15]  D. A. Bell,et al.  Information processing in scintigraphy: US Energy Research and Development Agency (undated). 209 pp. US $7.60 , 1977 .

[16]  D. Turner,et al.  An intuitive approach to receiver operating characteristic curve analysis. , 1978, Journal of nuclear medicine : official publication, Society of Nuclear Medicine.

[17]  C. Metz.  Basic principles of ROC analysis. , 1978, Seminars in nuclear medicine.

[18]  Applications of ROC analysis in diagnostic image evaluation , 1979 .

[19]  J. Swets.  ROC analysis applied to the evaluation of medical imaging techniques. , 1979, Investigative radiology.

[20]  Arthur G. Haus,et al.  The Physics of Medical Imaging: Recording System Measurements and Techniques , 1979 .

[21]  C. Metz,et al.  Statistical significance tests for binormal ROC curves , 1980 .

[22]  John A. Swets,et al.  Evaluation of diagnostic systems : methods from signal detection theory , 1982 .

[23]  J. Hanley,et al.  The meaning and use of the area under a receiver operating characteristic (ROC) curve. , 1982, Radiology.

[24]  J. Hanley,et al.  A method of comparing the areas under receiver operating characteristic curves derived from the same cases. , 1983, Radiology.

[25]  J. Hanley,et al.  Statistical Approaches to the Analysis of Receiver Operating Characteristic (ROC) Curves , 1984, Medical decision making : an international journal of the Society for Medical Decision Making.