A brief history of free-response receiver operating characteristic paradigm data analysis.

In the receiver operating characteristic paradigm the observer assigns a single rating to each image and the location of the perceived abnormality, if any, is ignored. In the free-response receiver operating characteristic paradigm the observer is free to mark and rate as many suspicious regions as are considered clinically reportable. Credit for a correct localization is given only if a mark is sufficiently close to an actual lesion; otherwise, the observer's mark is scored as a location-level false positive. Until fairly recently there existed no accepted method for analyzing the resulting relatively unstructured data containing random numbers of mark-rating pairs per image. This report reviews the history of work in this field, which has now spanned more than five decades. It introduces terminology used to describe the paradigm, proposed measures of performance (figures of merit), ways of visualizing the data (operating characteristics), and software for analyzing free-response receiver operating characteristic studies.

[1]  Darrin C. Edwards,et al.  Maximum likelihood fitting of FROC curves under an initial-detection-and-candidate-analysis model. , 2002, Medical physics.

[2]  Nancy A Obuchowski,et al.  A comparison of the Dorfman–Berbaum–Metz and Obuchowski–Rockette methods for receiver operating characteristic (ROC) data , 2005, Statistics in medicine.

[3]  Stephen L Hillis,et al.  Recent developments in the Dorfman-Berbaum-Metz procedure for multireader ROC study analysis. , 2008, Academic radiology.

[4]  K. Berbaum,et al.  Receiver operating characteristic rating analysis. Generalization to the population of readers and patients with the jackknife method. , 1992, Investigative radiology.

[5]  John F. Hamilton,et al.  A Free Response Approach To The Measurement And Characterization Of Radiographic Observer Performance , 1977, Other Conferences.

[6]  Andriy I. Bandos,et al.  On comparing methods for discriminating between actually negative and actually positive subjects with FROC type data. , 2008, Medical physics.

[7]  C. Rutter,et al.  Bootstrap estimation of diagnostic accuracy with patient-clustered data. , 2000, Academic radiology.

[8]  Stephen L Hillis,et al.  Monte Carlo validation of the Dorfman-Berbaum-Metz method using normalized pseudovalues and less data-based model simplification. , 2005, Academic radiology.

[9]  Ehsan Samei,et al.  The Handbook of Medical Image Perception and Techniques , 2009 .

[10]  Claudia Mello-Thoms,et al.  Spatial localization accuracy of radiologists in free-response studies: Inferring perceptual FROC curves from mark-rating data. , 2007, Academic radiology.

[11]  R. F. Wagner,et al.  Components-of-variance models and multiple-bootstrap experiments: an alternative method for random-effects, receiver operating characteristic analysis. , 2000, Academic radiology.

[12]  C. Metz,et al.  Visual detection and localization of radiographic images. , 1975, Radiology.

[13]  R. Swensson.  Unified measurement of observer performance in detecting and localizing target objects on images. , 1996, Medical physics.

[14]  Frank W. Samuelson,et al.  ADVANTAGES AND EXAMPLES OF RESAMPLING FOR CAD EVALUATION , 2007, 2007 4th IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[15]  D P Chakraborty,et al.  Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data. , 1989, Medical physics.

[16]  R. G. Fraser,et al.  Digital and conventional chest imaging: a modified ROC study of observer performance using simulated nodules. , 1986, Radiology.

[17]  Dev P Chakraborty,et al.  Validation and statistical power comparison of methods for analyzing free-response observer performance studies. , 2008, Academic radiology.

[18]  M Kallergi,et al.  Evaluating the performance of detection algorithms in digital mammography. , 1999, Medical physics.

[19]  H. Miller.  The FROC curve: a representation of the observer's performance for the method of free response. , 1969, The Journal of the Acoustical Society of America.

[20]  D. Chakraborty.  ROC curves predicted by a model of visual search , 2006, Physics in medicine and biology.

[21]  Stephen L Hillis,et al.  Power estimation for the Dorfman-Berbaum-Metz method. , 2004, Academic radiology.

[22]  D P Chakraborty.  A search model and figure of merit for observer data acquired according to the free-response paradigm. , 2006, Physics in medicine and biology.

[23]  D. Chakraborty,et al.  Free-response methodology: alternate analysis and a new observer-performance experiment. , 1990, Radiology.

[24]  P F Judy,et al.  Detection of noisy visual targets: Models for the effects of spatial uncertainty and signal-to-noise ratio , 1981, Perception & psychophysics.

[25]  Hong-Jun Yoon,et al.  Inverse dependence of search and classification performances in lesion localization tasks , 2012, Medical Imaging.

[26]  S. Hillis.  A comparison of denominator degrees of freedom methods for multiple observer ROC analysis , 2007, Statistics in medicine.

[27]  L T Niklason,et al.  Simulated pulmonary nodules: detection with dual-energy digital versus conventional radiography. , 1986, Radiology.

[28]  N A Obuchowski,et al.  Data analysis for detection and localization of multiple abnormalities with application to mammography. , 2000, Academic radiology.

[29]  Dev P Chakraborty,et al.  Observer studies involving detection and localization: modeling, analysis, and validation. , 2004, Medical physics.

[30]  David Gur,et al.  Localized detection and classification of abnormalities on FFDM and tomosynthesis examinations rated under an FROC paradigm. , 2011, AJR. American journal of roentgenology.

[31]  M F McEntee,et al.  On the choice of acceptance radius in free-response observer performance studies. , 2013, The British journal of radiology.

[32]  C. Metz,et al.  Comments on the generalization of receiver operating characteristic analysis to detection and localization tasks. , 1977, Physics in medicine and biology.

[33]  David Gur,et al.  Area under the Free‐Response ROC Curve (FROC) and a Related Summary Index , 2009, Biometrics.

[34]  D. Chakraborty,et al.  Recent developments in imaging system assessment methodology, FROC analysis and the search model. , 2011, Nuclear instruments & methods in physics research. Section A, Accelerators, spectrometers, detectors and associated equipment.

[35]  Kevin S. Berbaum,et al.  Satisfaction of search in diagnostic radiology. , 1989 .

[36]  Frank W. Samuelson,et al.  Comparing image detection algorithms using resampling , 2006, 3rd IEEE International Symposium on Biomedical Imaging: Nano to Macro, 2006..

[37]  James P. Egan,et al.  Operating Characteristics, Signal Detectability, and the Method of Free Response , 1961 .

[38]  L. Liberman,et al.  Breast imaging reporting and data system (BI-RADS). , 2002, Radiologic clinics of North America.

[39]  L. Popescu.  Nonparametric signal detectability evaluation using an exponential transformation of the FROC curve. , 2011, Medical physics.

[40]  D. Chakraborty.  Overview of the Receiver Operating Characteristic ( ROC ) Paradigm , 2011 .