Efficiency gain of paired split-plot designs in MRMC ROC studies

The widely used multi-reader multi-case ROC study design for comparing imaging modalities is the fullycrossed (FC) design: every reader reads every case of both modalities. In this work, we investigate the paired split-plot (PSP) designs that allow for reduced cost and increased flexibility compared to the FC design. In the PSP design, patient images from two modalities are read by the same readers, thereby the readings are paired across modalities. However, within each modality, not every reader reads every case. Instead, both the readers and the cases are partitioned into a number of groups and each group of readers read their own group of cases - a split-plot design. Using the U-statistic based variance analysis for AUC (i.e., area under the ROC curve), we show analytically that, with a fixed number of readings per reader, substantial statistical efficiency can be gained by the PSP design as compared to the FC design. Equivalently, we show that the PSP design can achieve the same statistical power as the FC design with substantially reduced number of readings. However, the efficiency/power gain of the PSP design comes with the increased cost of collecting a larger number of truthverified patient cases than the FC design. This means that one can trade off between different sources of cost and choose a least burdensome design. We demonstrate the advantages of the PSP design with a real-world reader study for the comparison of full field digital mammography with screen-film mammography.