Analysis of confidence level scores from an ROC study: comparison of three mammographic systems for detection of simulated calcifications

The purpose of this study is to compare the detection performance of three different mammography systems: screen/film (SF) combination, a-Si/CsI flat-panel (FP-), and charge-coupled device (CCD-) based systems. A 5-cm thick 50% adipose/50% glandular breast tissue equivalent slab phantom was used to provide an uniform background. Calcium carbonate grains of three different size groups were used to simulate microcalcifications (MCs): 112-125, 125-140, and 140-150 μm overlapping with the uniform background. Calcification images were acquired with the three mammography systems. Digital images were printed on hardcopy films. All film images were displayed on a mammographic viewer and reviewed by 5 mammographers. The visibility of the MC was rated with a 5-point confidence rating scale for each detection task, including the negative controls. Scores were averaged over all readers for various detectors and size groups. Receiver operating characteristic (ROC) analysis was performed and the areas under the ROC curves (Az’s) were computed for various imaging conditions. The results shows that (1) the FP-based system performed significantly better than the SF and CCD-based systems for individual size groups using ROC analysis (2) the FP-based system also performed significantly better than the SF and CCD-based systems for individual size groups using averaged confidence scale, and (3) the results obtained from the Az’s were largely correlated with these from confidence level scores. However, the correlation varied slightly among different imaging conditions.