Operating characteristics predicted by models for diagnostic tasks involving lesion localization.

In 1996 Swensson published an observer model that predicted receiver operating characteristic (ROC), localization ROC (LROC), free-response ROC (FROC) and alternative FROC (AFROC) curves, thereby achieving "unification" of different observer performance paradigms. More recently a model termed initial detection and candidate analysis (IDCA) has been proposed for fitting computer aided detection (CAD) generated FROC data, and recently a search model for human observer FROC data has been proposed. The purpose of this study was to derive IDCA and the search model based expressions for operating characteristics, and to compare the predictions to the Swensson model. For three out of four mammography CAD data sets all models yielded good fits in the high-confidence region, i.e., near the lower end of the plots. The search model and IDCA tended to better fit the data in the low-confidence region, i.e., near the upper end of the plots, particularly for FROC curves for which the Swensson model predictions departed markedly from the data. For one data set none of the models yielded satisfactory fits. A unique characteristic of search model and IDCA predicted operating characteristics is that the operating point is not allowed to move continuously to the lowest confidence limit of the corresponding Swensson model curves. This prediction is actually observed in the CAD raw data and it is the primary reason for the poor FROC fits of the Swensson model in the low-confidence region.

[1]  C E Metz,et al.  The "proper" binormal model: parametric receiver operating characteristic curve estimation with degenerate data. , 1997, Academic radiology.

[2]  C. Metz,et al.  Visual detection and localization of radiographic images. , 1975, Radiology.

[3]  J. Wolfe,et al.  Guided Search 2.0 A revised model of visual search , 1994, Psychonomic bulletin & review.

[4]  R. Swensson.  Unified measurement of observer performance in detecting and localizing target objects on images. , 1996, Medical physics.

[5]  C. Nodine,et al.  Using eye movements to study visual search and to improve tumor detection. , 1987, Radiographics : a review publication of the Radiological Society of North America, Inc.

[6]  Claudia Mello-Thoms,et al.  Spatial localization accuracy of radiologists in free-response studies: Inferring perceptual FROC curves from mark-rating data. , 2007, Academic radiology.

[7]  C. Metz,et al.  Comments on the generalization of receiver operating characteristic analysis to detection and localization tasks. , 1977, Physics in medicine and biology.

[8]  Charles E Metz,et al.  Receiver operating characteristic analysis: a tool for the quantitative evaluation of observer performance and imaging systems. , 2006, Journal of the American College of Radiology : JACR.

[9]  Darrin C. Edwards,et al.  Maximum likelihood fitting of FROC curves under an initial-detection-and-candidate-analysis model. , 2002, Medical physics.

[10]  Hiroshi Honda,et al.  Time Course of Satisfaction of Search , 1990 .

[11]  Berkman Sahiner,et al.  Evaluating computer-aided detection algorithms. , 2007, Medical physics.

[12]  D P Chakraborty,et al.  Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data. , 1989, Medical physics.

[13]  H L Kundel,et al.  A visual concept shapes image perception. , 1983, Radiology.

[14]  K S Berbaum,et al.  A contaminated binormal model for ROC data: Part II. A formal model. , 2000, Academic radiology.

[15]  D. Dorfman,et al.  Maximum-likelihood estimation of parameters of signal-detection theory and determination of confidence intervals—Rating-method data , 1969 .

[16]  T P Hutchinson.  Free-response operator characteristic models for visual search. , 2007, Physics in medicine and biology.

[17]  D. Chakraborty,et al.  Free-response methodology: alternate analysis and a new observer-performance experiment. , 1990, Radiology.

[18]  E. Conant,et al.  Holistic component of image perception in mammogram interpretation: gaze-tracking study. , 2007, Radiology.

[19]  D P Chakraborty.  A search model and figure of merit for observer data acquired according to the free-response paradigm. , 2006, Physics in medicine and biology.

[20]  John M. Irvine.  Assessing target search performance: the free-response operator characteristic model , 2004 .

[21]  James P. Egan,et al.  Operating Characteristics, Signal Detectability, and the Method of Free Response , 1961 .

[22]  P F Judy,et al.  Detection of noisy visual targets: Models for the effects of spatial uncertainty and signal-to-noise ratio , 1981, Perception & psychophysics.

[23]  Kevin S. Berbaum,et al.  Satisfaction of search in diagnostic radiology. , 1989 .

[24]  C. Metz.  ROC Methodology in Radiologic Imaging , 1986, Investigative radiology.

[25]  C. Metz,et al.  "Proper" Binormal ROC Curves: Theory and Maximum-Likelihood Estimation. , 1999, Journal of mathematical psychology.

[26]  Robert M Lewitt,et al.  Small nodule detectability evaluation using a generalized scan-statistic model , 2006, Physics in medicine and biology.

[27]  K. Berbaum,et al.  Proper receiver operating characteristic analysis: the bigamma model. , 1997, Academic radiology.

[28]  D. Chakraborty.  ROC curves predicted by a model of visual search , 2006, Physics in medicine and biology.