Performance and Robustness of Conrmatory Approaches

In this article, the performance of three conrmatory comparisons of means methods is inspected. The three types are hypothesis testing, model selection using information criteria, and Bayesian model selection. A simulation study is conducted to evaluate the performance of the three methods. For comparison, the performance of their exploratory counterparts are also determined. We demonstrate that conrmatory analyses have more power than exploratory analyses and that model selection has advantages over hypothesis testing. Little is known about the robustness of the dierent methods for violations of the assumptions, especially for conrmatory techniques. Therefore, we do another simulation study where we study the performance of the conrmatory methods when the homogeneity of variance assumption is violated. From this study, it can be concluded that the techniques are robust to heterogeneity when the sample sizes are equal. When the sample sizes are unequal, the performance is substantially aected by heterogeneity. However, the deviations from the baseline, where there is no heterogeneity, are not pronounced.