Information Systems and Health Care IV: Real-Time ROC Analysis to Evaluate Radiologists' Performanceof Interpreting Mammograpny

This paper describes how to use Receiver Operator Characteristic (ROC) analysis to evaluate radiologists’ performance of interpreting digital mammograms in real-time. We developed an experimental testing system, which implemented a set of clinical lesion-matching rules to prepare raw ROC data. The system can automatically provide detailed evaluations of the performance, such as sensitivity, specificity, positive predictive value, negative predictive value, diagnostic accuracy, ROC curve, and area under the curve (Az). Based on a preliminary evaluation of the system, we found that ROC analysis is appropriate for a real-time computer application, directly using the raw data from a database, to evaluate the performance of radiology residents.

[1]  Robert M. Nishikawa,et al.  Variations in measured performance of CAD schemes due to database composition and scoring protocol , 1998, Medical Imaging.

[2]  M. Zweig,et al.  Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. , 1993, Clinical chemistry.

[3]  C. Metz,et al.  Visual detection and localization of radiographic images. , 1975, Radiology.

[4]  J. Swets.  ROC analysis applied to the evaluation of medical imaging techniques. , 1979, Investigative radiology.

[5]  Helen C. Cowley,et al.  Improving the accuracy of mammography: volume and outcome relationships. , 2002, Journal of the National Cancer Institute.

[6]  Samuel G Armato,et al.  Automated detection of lung nodules in CT scans: effect of image reconstruction algorithm. , 2003, Medical physics.

[7]  D B Kopans,et al.  The accuracy of mammographic interpretation. , 1994, The New England journal of medicine.

[8]  J. Elmore,et al.  Accuracy of screening mammography using single versus independent double interpretation. , 2000, AJR. American journal of roentgenology.

[9]  G. P. Krestin,et al.  Detection of breast cancer with conventional mammography and contrast-enhanced MR imaging , 1998, European Radiology.

[10]  J Kettenbach,et al.  Effects of a microbubble contrast agent on breast tumors: computer-assisted quantitative assessment with color Doppler US--early experience. , 1998, Radiology.

[11]  Qiang Li,et al.  Selective enhancement filters for nodules, vessels, and airway walls in two- and three-dimensional CT scans. , 2003, Medical physics.

[12]  J M Lewin,et al.  Comparison of full-field digital mammography with screen-film mammography for cancer detection: results of 4,945 paired examinations. , 2001, Radiology.

[13]  C. Metz,et al.  "Proper" Binormal ROC Curves: Theory and Maximum-Likelihood Estimation. , 1999, Journal of mathematical psychology.

[14]  R. F. Wagner,et al.  Continuous versus categorical data for ROC analysis: some quantitative considerations. , 2001, Academic radiology.

[15]  C. Metz.  ROC Methodology in Radiologic Imaging , 1986, Investigative radiology.

[16]  H E Rockette,et al.  The use of continuous and discrete confidence judgments in receiver operating characteristic studies of diagnostic imaging techniques. , 1992, Investigative radiology.

[17]  R. Bird,et al.  Low-cost screening mammography: report on finances and review of 21,716 consecutive cases. , 1989, Radiology.

[18]  F. Hall,et al.  Nonpalpable breast lesions: recommendations for biopsy based on suspicion of carcinoma at mammography. , 1988, Radiology.

[19]  K Doi,et al.  Effect of case selection on the performance of computer-aided detection schemes. , 1994, Medical physics.

[20]  Min Wu,et al.  Online annotation tool for digital mammography1 , 2004 .

[21]  R. Rosenberg,et al.  Improvement in mammography interpretation skills in a community radiology practice after dedicated teaching courses: 2-year medical audit of 38,633 cases. , 1992, Radiology.

[22]  M. Giger,et al.  Improving breast cancer diagnosis with computer-aided diagnosis. , 1999, Academic radiology.

[23]  D B Spring,et al.  Evaluating the success of mammography at the local level: how to conduct an audit of your practice. , 1987, Radiologic clinics of North America.

[24]  E A Sickles,et al.  Quality assurance. How to audit your own mammography practice. , 1992, Radiologic clinics of North America.

[25]  Yuanshui Zheng,et al.  Online annotation tool for digital mammography. , 2004, Academic radiology.

[26]  K. Kerlikowske,et al.  Positive predictive value of screening mammography by age and family history of breast cancer. , 1993, JAMA.

[27]  N. Petrick,et al.  Computerized characterization of masses on mammograms: the rubber band straightening transform and texture analysis. , 1998, Medical physics.

[28]  P Y Poon,et al.  Medical audit of mammography: a simplified alternative. , 1992, Canadian Association of Radiologists journal = Journal l'Association canadienne des radiologistes.

[29]  L B Lusted,et al.  Signal detectability and medical decision-making. , 1971, Science.

[30]  K. Doi,et al.  Computer-aided diagnosis of pulmonary nodules: results of a large-scale observer test. , 1999, Radiology.

[31]  Nico Karssemeijer,et al.  Computer-Aided Diagnosis in Medical Imaging , 2001, IEEE Trans. Medical Imaging.

[32]  R. F. Wagner,et al.  Assessment of medical imaging and computer-assist systems: lessons from recent experience. , 2002, Academic radiology.