A Comparison of Parametric and Nonparametric Approaches to ROC Analysis of Quantitative Diagnostic Tests

Receiver operating characteristic (ROC) analysis, which yields indices of accuracy such as the area under the curve (AUC), is increasingly being used to evaluate the performances of diagnostic tests that produce results on continuous scales. Both par ametric and nonparametric ROC approaches are available to assess the discriminant capacity of such tests, but there are no clear guidelines as to the merits of each, particularly with non-binormal data. Investigators may worry that when data are non- Gaussian, estimates of diagnostic accuracy based on a binormal model may be dis torted. The authors conducted a Monte Carlo simulation study to compare the bias and sampling variability in the estimates of the AUCs derived from parametric and nonparametric procedures. Each approach was assessed in data sets generated from various configurations of pairs of overlapping distributions; these included the binormal model and non-binormal pairs of distributions where one or both pair members were mixtures of Gaussian (MG) distributions with different degrees of departures from bi- normality. The biases in the estimates of the AUCs were found to be very small for both parametric and nonparametric procedures. The two approaches yielded very close estimates of the AUCs and of the corresponding sampling variability even when data were generated from non-binormal models. Thus, for a wide range of distributions, concern about bias or imprecision of the estimates of the AUC should not be a major factor in choosing between the nonparametric and parametric approaches. Key words: ROC analysis; quantitative diagnostic test; comparison, parametric; binormal model; LABROC; nonparametric procedure; area under the curve (AUC). (Med Decis Making 1997;17:94-102)

[1]  James P. Egan,et al.  Signal detection theory and ROC analysis , 1975 .

[2]  J. Swets.  ROC analysis applied to the evaluation of medical imaging techniques. , 1979, Investigative radiology.

[3]  C. Metz.  ROC Methodology in Radiologic Imaging , 1986, Investigative radiology.

[4]  Mitchell H. Gail,et al.  A family of nonparametric statistics for comparing diagnostic markers with paired or unpaired data , 1989 .

[5]  N A Obuchowski,et al.  Computing Sample Size for Receiver Operating Characteristic Studies , 1994, Investigative radiology.

[6]  I Hinberg,et al.  Receiver operator characteristic (ROC) curves and non-normal data: an empirical study. , 1990, Statistics in medicine.

[7]  R M Centor,et al.  An Evaluation of Methods for Estimating the Area Under the Receiver Operating Characteristic (ROC) Curve , 1985, Medical decision making : an international journal of the Society for Medical Decision Making.

[8]  K Linnet,et al.  Comparison of quantitative diagnostic tests: type I error, power, and sample size. , 1987, Statistics in medicine.

[9]  J. Hanley.  The Robustness of the "Binormal" Assumptions Used in Fitting ROC Curves , 1988, Medical decision making : an international journal of the Society for Medical Decision Making.

[10]  R. Iman,et al.  Rank Transformations as a Bridge between Parametric and Nonparametric Statistics , 1981 .

[11]  D. M. Green,et al.  Signal detection theory and psychophysics , 1966 .

[12]  C. Begg,et al.  Advances in statistical methodology for diagnostic medicine in the 1980's. , 1991, Statistics in medicine.

[13]  D. Dorfman,et al.  Maximum-likelihood estimation of parameters of signal-detection theory and determination of confidence intervals—Rating-method data , 1969 .

[14]  E. DeLong,et al.  Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. , 1988, Biometrics.

[15]  J A Swets,et al.  Form of empirical ROCs in discrimination and diagnostic tasks: implications for theory and measurement of performance. , 1986, Psychological bulletin.

[16]  J. Hanley,et al.  The meaning and use of the area under a receiver operating characteristic (ROC) curve. , 1982, Radiology.

[17]  H E Rockette,et al.  Nonparametric estimation of degenerate ROC data sets used for comparison of imaging systems. , 1990, Investigative radiology.

[18]  Ronald L. Iman,et al.  The rank transformation as a method of discrimination with some examples , 1980 .

[19]  J. Swets.  Indices of discrimination or diagnostic accuracy: their ROCs and implied models. , 1986, Psychological bulletin.

[20]  R F Woolson,et al.  On the efficacy of the rank transformation in stepwise logistic and discriminant analysis. , 1993, Statistics in medicine.

[21]  D. Bamber.  The area above the ordinal dominance graph and the area below the receiver operating characteristic graph , 1975 .

[22]  C E Metz,et al.  Some practical issues of experimental design and data analysis in radiological ROC studies. , 1989, Investigative radiology.

[23]  J A Hanley,et al.  The use of the 'binormal' model for parametric ROC analysis of quantitative diagnostic tests. , 1996, Statistics in medicine.

[24]  J. Hanley.  Receiver operating characteristic (ROC) methodology: the state of the art. , 1989, Critical reviews in diagnostic imaging.

[25]  C B Begg,et al.  A General Regression Methodology for ROC Curve Estimation , 1988, Medical decision making : an international journal of the Society for Medical Decision Making.