Using Localization Data from Image Interpretations to Improve Estimates of Performance Accuracy

A recently developed model uses the localization of abnormalities on images to improve statistical precision in measuring detection accuracy Az , the area below an observer's receiver operating characteristic (ROC) curve for ratings of sampled normal and abnormal cases. This study evaluated that improvement by investigating how much the standard error of estimated Az decreased when the statistical analysis included localization data. Comparisons of analyses with vs without localizations were made for: 1) the estimates of Az from observers' rating ROC curves for nodular lesions on clinical chest films and liver CT scans; 2) the probability of correct choices between paired samples of normal and abnormal cases (equivalent to Az ); and 3) the sampling distributions of Az measured in Monte Carlo simulations of 2,000 independent rating experiments. Localization information considerably improved the precision of Az estimates, particularly when detection accuracy was low (Az ~ 0.60). These data provided roughly the same benefits in estimation precision as would two-to-fourfold increases in the sizes of both 1) the samples of positive and negative cases and 2) the observer samples used to estimate Az means. Key words: ROC analysis; LROC; observer performance accuracy; observer vanability; detection; localization; image interpretation. (Med Decis Making 2000;20:170-185)

[1]  Jill L. King,et al.  Using incomplete and imprecise localization data on images to improve estimates of detection accuracy , 1999, Medical Imaging.

[2]  Philip F. Judy,et al.  Nodule polarity effects on detection and localization performance in liver CT images , 1997, Medical Imaging.

[3]  R. Swensson.  Unified measurement of observer performance in detecting and localizing target objects on images. , 1996, Medical physics.

[4]  P F Judy,et al.  Visualization and detection-localization on computed tomographic images. , 1991, Investigative radiology.

[5]  C E Metz,et al.  Some practical issues of experimental design and data analysis in radiological ROC studies. , 1989, Investigative radiology.

[6]  J. Hanley,et al.  The meaning and use of the area under a receiver operating characteristic (ROC) curve. , 1982, Radiology.

[7]  C. Metz,et al.  Visual detection and localization of radiographic images. , 1975, Radiology.

[8]  D. Dorfman,et al.  Maximum-likelihood estimation of parameters of signal-detection theory and determination of confidence intervals—Rating-method data , 1969 .

[9]  C. Metz,et al.  Statistical significance tests for binormal ROC curves , 1980 .

[10]  R G Swensson,et al.  Search and nonsearch protocols for radiographic consultation. , 1990, Radiology.

[11]  C E Floyd,et al.  Artificial Neural Networks for Single Photon Emission Computed Tomography: A Study of Cold Lesion Detection and Localization , 1993, Investigative radiology.

[12]  Byron J. T. Morgan,et al.  Some aspects of ROC curve-fitting: Normal and logistic models , 1972 .

[13]  N A Obuchowski,et al.  Computing Sample Size for Receiver Operating Characteristic Studies , 1994, Investigative radiology.

[14]  P F Judy,et al.  Detection of noisy visual targets: Models for the effects of spatial uncertainty and signal-to-noise ratio , 1981, Perception & psychophysics.

[15]  K Doi,et al.  Improvement in radiologists' detection of clustered microcalcifications on mammograms. The potential of computer-aided diagnosis. , 1990, Investigative radiology.

[16]  D. Bamber.  The area above the ordinal dominance graph and the area below the receiver operating characteristic graph , 1975 .

[17]  J. D. Haberman,et al.  Analysis of mammography: a blind interpretation of BCDDP radiographs. , 1983, Radiology.