Comparing observer performance with mixture distribution analysis when there is no external gold standard

Mixture distribution analysis (MDA) is proposed as a statistical methodology for comparing observer readings on different imaging modalities when the image findings cannot be independently verified. The study utilized a data set consisting of independent, blinded readings by 4 radiologists of a stratified sample of 95 bedside chest images obtained using computed radiography. Each case was rad on hard and soft copy. The area under the ROC curve (AUC) was calculated using ROCFIT and the relative percent correct (RPC) was calculated from point distributions estimated by the MDA. The expectation maximization algorithm was used to perform a maximum likelihood estimation of the fit to either 3, 4 or 5 point distributions. There was agreement between the AUC and the RPC based upon 3 point distributions representing easy normals, hard normals and abnormals, easy abnormals, hard normals, hard abnormals and easy abnormals. We conclude that the MDA may be a viable alternative to the ROC for evaluating observer performance on imaging modalities in clinical settings where image verification is either difficult or impossible.