Methods for dealing with multiple outcomes in meta-analysis: a comparison between averaging effect sizes, robust variance estimation and multilevel meta-analysis

Abstract This study investigates three methods to handle dependency among effect size estimates in meta-analysis arising from studies reporting multiple outcome measures taken on the same sample. The three-level approach is compared with the method of robust variance estimation, and with averaging effects within studies. A simulation study is performed, and the fixed and random effect estimates of the three methods are compared with each other. Both the robust variance estimation and three-level approach result in unbiased estimates of the fixed effects, corresponding standard errors and variances. Averaging effect sizes results in overestimated standard errors when the effect sizes within studies are truly independent. Although the robust variance and three-level approach are more complicated to use, they have the advantage that they do not require an estimate of the correlation between outcomes, and they still result in unbiased parameter estimates.

[1]  S. Vaughn,et al.  A Synthesis of Reading Interventions and Effects on Reading Comprehension Outcomes for Older Struggling Readers , 2009, Review of educational research.

[2]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[3]  Wim Van den Noortgate,et al.  Meta-analysis of multiple outcomes: a multilevel approach , 2014, Behavior Research Methods.

[4]  Zachary F. Fisher,et al.  robumeta: An R-package for robust variance estimation in meta-analysis , 2015, 1503.02220.

[5]  L. Hedges,et al.  Introduction to Meta‐Analysis , 2009, International Coaching Psychology Review.

[6]  Betsy Jane Becker,et al.  Modeling multivariate effect sizes. , 1988 .

[7]  Betsy Jane Becker,et al.  Multivariate meta-analysis. , 2000 .

[8]  Larry V. Hedges,et al.  Correcting a Significance Test for Clustering , 2007 .

[9]  Nicholas D. Myers,et al.  A Review of Meta-Analyses in Education , 2012 .

[10]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[11]  H. Engelhardt,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods.Anthony S. Bryk , Stephen W. Raudenbush , 1994 .

[12]  Patrick Onghena,et al.  Multilevel Meta-Analysis: A Comparison with Traditional Meta-Analytical Procedures , 2003 .

[13]  William R. Jenson,et al.  Statistical reform: Evidence‐based practice, meta‐analyses, and single subject designs , 2007 .

[14]  M. Cheung.  Three‐level meta‐analysis , 2015 .

[15]  Elizabeth Tipton,et al.  Robust variance estimation with dependent effect sizes: practical considerations including a software tutorial in Stata and spss , 2014, Research synthesis methods.

[16]  Steven D. Brown,et al.  Handbook of applied multivariate statistics and mathematical modeling , 2000 .

[17]  Ingram Olkin,et al.  Stochastically dependent effect sizes. , 1994 .

[18]  A. Boomsma,et al.  Robustness Studies in Covariance Structure Modeling , 1998 .

[19]  Steven A. Stahl,et al.  Systematic Phonics Instruction Helps Students Learn to Read: Evidence from the National Reading Panel’s Meta-Analysis , 2001 .

[20]  C S Berkey,et al.  Multiple-outcome meta-analysis of clinical trials. , 1996, Statistics in medicine.

[21]  Jerome V. D'Agostino,et al.  A Meta-Analysis of Reading Recovery in United States Schools , 2004 .

[22]  Spyros Konstantopoulos,et al.  Fixed effects and variance components estimation in three‐level meta‐analysis , 2011, Research synthesis methods.

[23]  Thomas D. Cook,et al.  Threats to the validity of research syntheses , 1994 .

[24]  Elizabeth Tipton,et al.  Robust variance estimation in meta‐regression with dependent effect size estimates , 2010, Research synthesis methods.

[25]  S. Raudenbush,et al.  A multivariate mixed linear model for meta-analysis. , 1996 .

[26]  Kerrie Mengersen,et al.  Multivariate meta‐analysis , 2003, Statistics in medicine.

[27]  Mike W-L Cheung,et al.  Modeling dependent effect sizes with three-level meta-analyses: a structural equation modeling approach. , 2014, Psychological methods.

[28]  Tom A. B. Snijders,et al.  Multilevel Analysis , 2011, International Encyclopedia of Statistical Science.

[29]  Elizabeth Tipton,et al.  Small sample adjustments for robust variance estimation with meta-regression. , 2015, Psychological methods.

[30]  Wim Van den Noortgate,et al.  Three-level meta-analysis of dependent effect sizes , 2012, Behavior Research Methods.

[31]  Jeffrey C. Valentine,et al.  Research Synthesis and Meta-Analysis , 2008 .

[32]  Xiao-Hua Zhou,et al.  Statistical Methods for Meta‐Analysis , 2008 .

[33]  Shu Fai Cheung,et al.  Dependent effect sizes in meta-analysis: incorporating the degree of interdependence. , 2004, The Journal of applied psychology.

[34]  J. Norris,et al.  Research synthesis and meta‐analysis , 2012 .

[35]  Mark W. Lipsey,et al.  Practical Meta-Analysis , 2000 .

[36]  M. S. Patel,et al.  An introduction to meta-analysis. , 1989, Health Policy.

[37]  Richard I. Parker,et al.  Evaluating Single-Case Research Data: A Comparison of Seven Statistical Methods , 2003 .

[38]  Rumen Manolov,et al.  A comparison of mean phase difference and generalized least squares for analyzing single-case data. , 2013, Journal of school psychology.

[39]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .