Analysis of double reading in an observer study

Previously we showed based on theoretical analysis that it is possible to attain greater diagnostic performance from appropriately combining the diagnostic opinions of two or more equally skilled readers. Such gain in performance is available from combining the readers' "latent decision variables" that are accessible through ROC analysis, but is generally ambiguous at best if the readers' binary decisions with regard to clinical actions (e.g., recall vs. annual screening mammogram) are combined. We now analyze the data of an observer study. In this observer study, ten radiologists interpreted 104 cases of mammograms containing clustered microcalcifications in a diagnostic-study setting to decide whether to recommend biopsy. They also reported diagnostic confidence on a quasi-continuous scale that the calcifications indicated malignancy. A previous analysis showed that combining the radiologists' binary decisions (biopsy vs. no biopsy) would change both sensitivity and specificity generally along the radiologists' single-reading, average, ROC curve but would not increase the diagnostic performance. Combining two radiologists' "latent decision variables" resulted in small increases in the ROC curves consistent with the theoretical predictions. However, the shapes of the single-reading ROC curves were inconsistent with the expectation of the clinical diagnostic-study setting because all benign cases in the observer study were difficult-to-diagnose cases (all cases clinically biopsied). The double-reading results would have been different, and gains in diagnostic performance possible, if the ROC curve shape more accurately resembled that of clinical practice. There is need to estimate the ROC curve of clinical practice.