Prevalence effect in a laboratory environment.

PURPOSE To measure observer performance at various levels of prevalence. MATERIALS AND METHODS A multiobserver multiabnormality receiver operating characteristic (ROC) study to assess the effect of prevalence on observer performance was conducted. Fourteen observers, including eight faculty members, two fellows, and four residents, interpreted 1,632 posteroanterior chest images with five prevalence levels by using a nested study design. Performance comparisons were accomplished by using a multireader multicase approach to assess the effect of prevalence from 28% (69 of 249) to 2% (31 of 1,577) on diagnostic accuracy. The mean times required to review and report a case were analyzed and compared for different levels of prevalence and readers' experience. RESULTS Area under the ROC curve demonstrated that, with the study experimental conditions, no significant effect could be measured as a function of prevalence (P >.05) for any abnormality, group of cases, or readers. There were no significant differences (P >.05) in the mean times required to review and report cases at different prevalence levels and with different groups of readers. CONCLUSION The consistency in the results and the size of this study suggest that with laboratory conditions, if a prevalence effect exists, it is quite small in magnitude; hence, it will not likely alter conclusions derived from such studies.

[1]  R G Swensson,et al.  The value of searching films without specific preconceptions. , 1985, Investigative radiology.

[2]  C. Metz.  ROC Methodology in Radiologic Imaging , 1986, Investigative radiology.

[3]  C A Kelsey,et al.  Factors affecting perception of pulmonary lesions. , 1983, Radiologic clinics of North America.

[4]  John W. Tukey,et al.  Efficient Utilization of Non-Numerical Information in Quantitative Analysis General Theory and the Case of Simple Order , 1963 .

[5]  Harold L. Kundel,et al.  Disease prevalence and the index of detectability: a survey of studies of lung cancer detection by chest radiography , 2000, Medical Imaging.

[6]  H E Rockette,et al.  Receiver operating characteristic analysis of chest image interpretation with conventional, laser-printed, and high-resolution workstation images. , 1990, Radiology.

[7]  K. Doi,et al.  Digital radiography of subtle pulmonary abnormalities: an ROC study of the effect of pixel size on observer performance. , 1986, Radiology.

[8]  K. Berbaum,et al.  Receiver operating characteristic rating analysis. Generalization to the population of readers and patients with the jackknife method. , 1992, Investigative radiology.

[9]  E. B. Page.  Ordered Hypotheses for Multiple Treatments: A Significance Test for Linear Ranks , 1963 .

[10]  D. M. Green,et al.  Signal detection theory and psychophysics , 1966 .

[11]  H E Rockette,et al.  Effects of luminance and resolution on observer performance with chest radiographs. , 2000, Radiology.

[12]  C E Metz,et al.  Some practical issues of experimental design and data analysis in radiological ROC studies. , 1989, Investigative radiology.

[13]  David Manning,et al.  Effects of prevalence on visual search and decision making in fracture detection , 2001, SPIE Medical Imaging.

[14]  John A. Swets,et al.  Evaluation of diagnostic systems : methods from signal detection theory , 1982 .

[15]  C A Britton,et al.  Digital radiography and conventional imaging of the chest: a comparison of observer performance. , 1994, AJR. American journal of roentgenology.

[16]  A R Feinstein,et al.  Context bias. A problem in diagnostic radiology. , 1996, JAMA.

[17]  N. Obuchowski,et al.  Hypothesis testing of diagnostic accuracy for multiple readers and multiple tests: An anova approach with dependent observations , 1995 .

[18]  S. Zeger,et al.  Longitudinal data analysis using generalized linear models , 1986 .

[19]  H L Kundel,et al.  Disease prevalence and radiological decision making. , 1982, Investigative Radiology.

[20]  N A Obuchowski,et al.  Multireader, multimodality receiver operating characteristic curve studies: hypothesis testing and sample size estimation using an analysis of variance approach with dependent observations. , 1995, Academic radiology.

[21]  F A DiBianca,et al.  The effect of varying spatial resolution on the detectability of diffuse pulmonary nodules. Assessment with digitized conventional radiographs. , 1981, Radiology.