Pairwise Multiple Comparison Tests when Data are Nonnormal

Numerous authors suggest that the data gathered by investigators are not normal in shape. Accordingly, methods for assessing pairwise multiple comparisons of means with traditional statistics will frequently result in biased rates of Type I error and depressed power to detect effects. One solution is to obtain a critical value to assess statistical significance through bootstrap methods. The SAS system can be used to conduct step-down bootstrapped tests. The authors investigated this approach when data were neither normal in form nor equal in variability in balanced and unbalanced designs. They found that the step-down bootstrap method resulted in substantially inflated rates of error when variances and group sizes were negatively paired. Based on their results, and those reported elsewhere, the authors recommend that researchers should use trimmed means and Winsorized variances with a heteroscedastic test statistic. When group sizes are equal, the bootstrap procedure effectively controlled Type I error rates.