Case sampling in LROC: a Monte Carlo analysis

We conducted a series of Monte Carlo simulations to investigate how hypothesis testing for modality effects in multireader localization ROC (LROC) studies is influenced by case effects. One specific goal was to evaluate for LROC studies the Dorfman-Berbaum-Metz method of analyzing case effects in reader data acquired from a single case-set. Previous evaluations with ROC study simulations found the DBM method to be moderately conservative. Our simulations, using procedures adapted from those earlier works, showed the DBM method to be a conservative test of modality effect in LROC studies as well. The degree of conservatism was greater for a critical value of (alpha) equals0.05 than for (alpha) equals0.01, and was not moderated by increased numbers of readers or cases. Other simulations investigated the tradeoff between power and empirical type-I error rate for the DBM method and two standard hypothesis tests. Besides the DBM method, a two-way analysis of variance (ANOVA) was applied to performance indices based on the LROC curve under an assumption of negligible case effects. The third test was a three-way ANOVA applied to performance indices, which required two sets of images per modality. With (alpha) equals0.01, the DBM method outperformed the other tests for studies with low numbers of readers and cases. In most other situations, its performance lagged behind that of the other tests.