Small-Sample Adjustments for Tests of Moderators and Model Fit Using Robust Variance Estimation in Meta-Regression

Meta-analyses often include studies that report multiple effect sizes based on a common pool of subjects or that report effect sizes from several samples that were treated with very similar research protocols. The inclusion of such studies introduces dependence among the effect size estimates. When the number of studies is large, robust variance estimation (RVE) provides a method for pooling dependent effects, even when information on the exact dependence structure is not available. When the number of studies is small or moderate, however, test statistics and confidence intervals based on RVE can have inflated Type I error. This article describes and investigates several small-sample adjustments to F-statistics based on RVE. Simulation results demonstrate that one such test, which approximates the test statistic using Hotelling’s T2 distribution, is level-α and uniformly more powerful than the others. An empirical application demonstrates how results based on this test compare to the large-sample F-test.

[1]  E. Hedberg.  ROBUMETA: Stata module to perform robust variance estimation in meta-regression with dependent effect size estimates , 2011 .

[2]  Elizabeth Tipton,et al.  Small sample adjustments for robust variance estimation with meta-regression. , 2015, Psychological methods.

[3]  L. Hedges,et al.  The Handbook of Research Synthesis and Meta-Analysis , 2009 .

[4]  R. Bell,et al.  Improved hypothesis testing for coefficients in generalized estimating equations with small samples of clusters , 2006, Statistics in medicine.

[5]  R. Muirhead.  Aspects of Multivariate Statistical Theory , 1982, Wiley Series in Probability and Statistics.

[6]  Jin-Ting Zhang.  Tests of Linear Hypotheses in the ANOVA under Heteroscedasticity , 2013 .

[7]  James G. MacKinnon,et al.  Wild Bootstrap Inference for Wildly Different Cluster Sizes , 2017 .

[8]  Elizabeth Tipton,et al.  Robust variance estimation in meta‐regression with binary dependent effects , 2013, Research synthesis methods.

[9]  Joshua R Polanin,et al.  The use of meta‐analytic statistical significance testing , 2015, Research synthesis methods.

[10]  Sandra Jo Wilson,et al.  The comparative effectiveness of outpatient treatment for adolescent substance abuse: a meta-analysis. , 2013, Journal of substance abuse treatment.

[11]  Tiina Ojanen,et al.  Social Goals and Youth Aggression: Meta-Analysis of Prosocial and Antisocial Goals. , 2012 .

[12]  Elizabeth Tipton,et al.  Mindfulness Based Stress Reduction (MBSR) for Improving Health, Quality of Life, and Social Functioning in Adults , 2012 .

[13]  K. Krishnamoorthy,et al.  Modified Nel and Van der Merwe test for the multivariate Behrens–Fisher problem , 2004 .

[14]  G. W. Hill,et al.  Algorithm 396: Student's t-quantiles , 1970, CACM.

[15]  J. Norris,et al.  Research synthesis and meta‐analysis , 2012 .

[16]  Jin-Ting Zhang,et al.  An Approximate Hotelling T^2-Test for Heteroscedastic One-Way MANOVA , 2012 .

[17]  Ryan T. Williams,et al.  Using robust standard errors to combine multiple regression estimates with meta-analysis , 2012 .

[18]  R. Bell,et al.  GENERALIZATIONS OF BIASED REDUCED LINEARIZATION , 2001 .

[19]  D. Uttal,et al.  The malleability of spatial skills: a meta-analysis of training studies. , 2013, Psychological bulletin.

[20]  Nicholas D. Myers,et al.  A Review of Meta-Analyses in Education , 2012 .

[21]  Ralph A. Alexander,et al.  A New and Simpler Approximation for ANOVA Under Variance Heterogeneity , 1994 .

[22]  Ingram Olkin,et al.  Stochastically dependent effect sizes. , 1994 .

[23]  Zachary F. Fisher,et al.  robumeta: An R-package for robust variance estimation in meta-analysis , 2015, 1503.02220.

[24]  Mark W. Lipsey,et al.  Dropout Prevention and Intervention Programs: Effects on School Completion and Dropout Among School-Aged Children and Youth: A Systematic Review , 2010 .

[25]  D. G. Nel,et al.  A solution to the multivariate behrens-fisher problem , 1986 .

[26]  Xiaohong Chen,et al.  Recent advances and future directions in causality, prediction, and specification analysis , 2013 .

[27]  Elizabeth Tipton,et al.  Robust variance estimation with dependent effect sizes: practical considerations including a software tutorial in Stata and spss , 2014, Research synthesis methods.

[28]  P. Cornelius,et al.  Approximate F-tests of multiple degree of freedom hypotheses in generalized least squares analyses of unbalanced split-plot experiments , 1996 .

[29]  W. Pan,et al.  Small‐sample adjustments in using the sandwich variance estimator in generalized estimating equations , 2002, Statistics in medicine.

[30]  James G. MacKinnon,et al.  Thirty Years of Heteroskedasticity-Robust Inference , 2013 .

[31]  A. Hayes,et al.  A New Test of Linear Hypotheses in OLS Regression Under Heteroscedasticity of Unknown Form , 2008 .

[32]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .

[33]  Elizabeth Tipton,et al.  Robust variance estimation in meta‐regression with dependent effect size estimates , 2010, Research synthesis methods.

[34]  Daniel F. McCaffrey,et al.  Bias reduction in standard errors for linear regression with multi-stage samples , 2002 .