ROBUSTNESS IN META-ANALYSIS: AN EMPIRICAL COMPARISON OF POINT AND INTERVAL ESTIMATES OF STANDARDIZED MEAN DIFFERENCES AND CLIFF’S DELTA

With the growing popularity of meta-analytic techniques to analyze and synthesize results across sets of empirical studies, have come concerns about the sensitivity of traditional tests in meta-analysis to violations of assumptions. This is particularly distressing because the tenability of such assumptions in primary studies is often impossible to evaluate unless sufficient details are reported. Robust estimates of effect size, such as Cliff’s δ , may yield superior inferences about the population effect size. The purpose of this study was to compare standardized mean differences (Cohen’s d and Hedges’ g) and δ in terms of the accuracy and precision of interval estimates of population mean effect size in meta-analysis. Factors investigated in the Monte Carlo study included characteristics of both the populations from which samples were drawn (distribution shape, variance heterogeneity, and population effect size) and the corpus of studies in each meta-analysis (sample size and number of studies). Although d and g provided relatively unbiased estimates when the assumptions were met, inferences with δ evidenced less bias under nonnormality and variance heterogeneity.