Selection of subtle cases for observer-performance studies: the importance of knowing the true diagnosis.

RATIONALE AND OBJECTIVES To assess the usefulness of classifying degree of difficulty in abnormality detection and to determine the effect of knowing the true diagnosis when selecting subtle images for observer-performance studies. MATERIALS AND METHODS A total of 529 posteroanterior chest images that had been used in a multiabnormality, multireader observer-performance study were rated by three observers as to the difficulty of determining the presence or absence of each abnormality when the true diagnosis was known and when it was not known. Changes in image subtlety ratings were evaluated, and actual observer-performance results for the different groups of images grouped according to raters' classifications with and without availability of the true diagnosis were compared. RESULTS The majority of negative cases (9,168 of 12,258, 74.8%) were rated as "easy" to determine. Substantial changes were made during the selection of the "subtle" case category when the truth was known compared with when the truth was not provided. These changes caused differences between typical and subtle cases in terms of observer performance. Combined ratings of case subtlety by agreement of multiple classifiers resulted in a well-ordered selection with decreasing observer performance as a function of subtlety ratings. CONCLUSION Cases for observer-performance studies that stress the diagnostic system can be successfully selected in the multiple-disease setting by experienced readers and should be selected with the truth known to the raters. The degree of agreement by multiple raters can be used to refine subtlety ratings.