Degeneracy and discrete receiver operating characteristic rating data.

RATIONALE AND OBJECTIVES Observer performance studies sometimes use too few cases for estimating diagnostic accuracy from binormal receiver operating characteristic (ROC) curves. One important problem is degenerate data sets. We compared a new algorithm, RSCORE4, with the exact-solution approach to degeneracy in ROCFIT and with the Wilcoxon statistic. METHODS Degenerate ROC solutions result from empty cells in the data matrix. We addressed this problem by adding a small constant to empty cells in a maximum-likelihood program, RSCORE4. When this method failed, the program branched to a pattern-search algorithm. We tested the program in a series of Monte Carlo studies. RESULTS RSCORE4 converged to nondegenerate solutions in every case and gave results closer to population values than ROCFIT or Wilcoxon. ROCFIT converged to exact-fit degenerate solutions, those with zero or infinite parameter values, in more than 40% of the samples. The Wilcoxon statistic was biased. CONCLUSION RSCORE4 seems to outperform other currently recommended methods for dealing with degeneracy.

[1]  H E Rockette,et al.  Nonparametric estimation of degenerate ROC data sets used for comparison of imaging systems. , 1990, Investigative radiology.

[2]  E. L. Lehmann,et al.  Theory of point estimation , 1950 .

[3]  C E Metz,et al.  Some practical issues of experimental design and data analysis in radiological ROC studies. , 1989, Investigative radiology.

[4]  Donald D. Dorfman,et al.  Estimation of signal detection theory parameters from rating-method data: A comparison of the method of scoring and direct search* , 1973 .

[5]  H E Rockette,et al.  On the generalization of the receiver operating characteristic analysis to the population of readers and cases with the jackknife method: an assessment. , 1995, Academic radiology.

[6]  R. Serfling.  Approximation Theorems of Mathematical Statistics , 1980 .

[7]  A. Agresti,et al.  Categorical Data Analysis , 1991, International Encyclopedia of Statistical Science.

[8]  K. Berbaum,et al.  Receiver operating characteristic rating analysis. Generalization to the population of readers and patients with the jackknife method. , 1992, Investigative radiology.

[9]  Robert Hooke,et al.  `` Direct Search'' Solution of Numerical and Statistical Problems , 1961, JACM.

[10]  D. Dorfman,et al.  Maximum-likelihood estimation of parameters of signal-detection theory and determination of confidence intervals—Rating-method data , 1969 .

[11]  John A. Nelder,et al.  A Simplex Method for Function Minimization , 1965, Comput. J..

[12]  T. Brubaker,et al.  Nonlinear Parameter Estimation , 1979 .

[13]  J. Hanley,et al.  Statistical Approaches to the Analysis of Receiver Operating Characteristic (ROC) Curves , 1984, Medical decision making : an international journal of the Society for Medical Decision Making.

[14]  John A. Swets,et al.  Evaluation of diagnostic systems : methods from signal detection theory , 1982 .