Recent advances in observer performance methodology: jackknife free-response ROC (JAFROC).

The jackknife free-response receiver operating characteristic (JAFROC) method allows quantitative analysis of observer data such as that observed when radiologists interpret images, which could contain more than one lesion and a location can be reported for each perceived lesion. The method was recently validated with a perception-based simulation model that incorporated the detectability parameter of the standard binormal ROC model, and in addition allowed simultaneous samples from both noise and signal distributions. The total number of noise samples is an important new parameter that measures reader expertise. The new sampling model incorporates search, which is an integral part of lesion detection that has not been possible to model until now. The model was used to generate simulated FROC ratings data, which was used to assess the statistical validity of JAFROC analysis. We found that JAFROC analysis is a statistically valid approach for analysing FROC data and that JAFROC analysis exhibited significantly greater statistical power than the existing ROC approach.

[1]  K. Berbaum,et al.  Receiver operating characteristic rating analysis. Generalization to the population of readers and patients with the jackknife method. , 1992, Investigative radiology.

[2]  D. Chakraborty,et al.  Free-response methodology: alternate analysis and a new observer-performance experiment. , 1990, Radiology.

[3]  Miguel P Eckstein,et al.  Classification image analysis: estimation and statistical inference for two-alternative forced-choice experiments. , 2002, Journal of vision.

[4]  Dev P. Chakraborty.  Proposed solution to the FROC problem and an invitation to collaborate , 2003, SPIE Medical Imaging.

[5]  John F. Hamilton,et al.  A Free Response Approach To The Measurement And Characterization Of Radiographic Observer Performance , 1977, Other Conferences.

[6]  C. Metz.  ROC Methodology in Radiologic Imaging , 1986, Investigative radiology.

[7]  C. Metz.  Basic principles of ROC analysis. , 1978, Seminars in nuclear medicine.

[8]  A. Burgess.  Comparison of receiver operating characteristic and forced choice observer performance measurement methods. , 1995, Medical physics.

[9]  Matthew A. Kupinski,et al.  Ideal-Observer Performance under Signal and Background Uncertainty , 2003, IPMI.

[10]  D P Chakraborty,et al.  Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data. , 1989, Medical physics.

[11]  C E Metz,et al.  Variance-component modeling in the analysis of receiver operating characteristic index estimates. , 1997, Academic radiology.

[12]  Dev Chakraborty,et al.  Statistical power in observer-performance studies: comparison of the receiver operating characteristic and free-response methods in tasks involving localization. , 2002, Academic radiology.

[13]  Dev P Chakraborty,et al.  Observer studies involving detection and localization: modeling, analysis, and validation. , 2004, Medical physics.

[14]  D. Dorfman,et al.  Maximum-likelihood estimation of parameters of signal-detection theory and determination of confidence intervals—Rating-method data , 1969 .

[15]  C A Roe,et al.  Dorfman-Berbaum-Metz method for statistical analysis of multireader, multimodality receiver operating characteristic data: validation with computer simulation. , 1997, Academic radiology.

[16]  R. Swensson.  Unified measurement of observer performance in detecting and localizing target objects on images. , 1996, Medical physics.

[17]  C E Metz,et al.  Some practical issues of experimental design and data analysis in radiological ROC studies. , 1989, Investigative radiology.

[18]  Dev P. Chakraborty,et al.  Jackknife free-response ROC methodology , 2004, SPIE Medical Imaging.

[19]  C. Nodine,et al.  Using eye movements to study visual search and to improve tumor detection. , 1987, Radiographics : a review publication of the Radiological Society of North America, Inc.

[20]  N A Obuchowski,et al.  Data analysis for detection and localization of multiple abnormalities with application to mammography. , 2000, Academic radiology.