Meta-Analysis With Complex Research Designs

Previous research has shown that treating dependent effect sizes as independent inflates the variance of the mean effect size and introduces bias by giving studies with more effect sizes more weight in the meta-analysis. This article summarizes the different approaches to handling dependence that have been advocated by methodologists, some of which are more feasible to implement with education research studies than others. A case study using effect sizes from a recent meta-analysis of reading interventions is presented to compare the results obtained from different approaches to dealing with dependence. Overall, mean effect sizes and variance estimates were found to be similar, but estimates of indexes of heterogeneity varied. Meta-analysts are advised to explore the effect of the method of handling dependence on the heterogeneity estimates before conducting moderator analyses and to choose the approach to dependence that is best suited to their research question and their data set.

[1]  Elizabeth Tipton,et al.  Robust variance estimation in meta‐regression with dependent effect size estimates , 2010, Research synthesis methods.

[2]  L. Hedges,et al.  Multiple Outcomes or Time‐Points within a Study , 2009, Introduction to Meta‐Analysis.

[3]  Kerrie Mengersen,et al.  Multivariate meta‐analysis , 2003, Statistics in medicine.

[4]  Donald B. Rubin,et al.  Meta-Analytic Procedures for Combining Studies With Multiple Effect Sizes , 1986 .

[5]  Noel A. Card.  Applied Meta-Analysis for Social Science Research , 2011 .

[6]  S. Vaughn,et al.  A Meta-Analysis of Interventions for Struggling Readers in Grades 4–12 , 2015, Journal of learning disabilities.

[7]  Mike W-L Cheung,et al.  Modeling dependent effect sizes with three-level meta-analyses: a structural equation modeling approach. , 2014, Psychological methods.

[8]  D. Uttal,et al.  The malleability of spatial skills: a meta-analysis of training studies. , 2013, Psychological bulletin.

[9]  Loan Tran,et al.  A Meta-Analysis of the RTI Literature for Children at Risk for Reading Disabilities , 2011, Journal of learning disabilities.

[10]  B. Becker,et al.  The Degree of Dependence Between Multiple-Treatment Effect Sizes , 2010, Multivariate behavioral research.

[11]  Jonathan R. Flojo,et al.  Mathematics Instruction for Students With Learning Disabilities: A Meta-Analysis of Instructional Components , 2009 .

[12]  Mike W.-L. Cheung,et al.  Fixed-Effects Meta-Analyses as Multiple-Group Structural Equation Models , 2010 .

[13]  Jerome V. D'Agostino,et al.  A Meta-Analysis of Reading Recovery in United States Schools , 2004 .

[14]  Spyros Konstantopoulos,et al.  Fixed effects and variance components estimation in three‐level meta‐analysis , 2011, Research synthesis methods.

[15]  Tiina Ojanen,et al.  Social Goals and Youth Aggression: Meta-Analysis of Prosocial and Antisocial Goals. , 2012 .

[16]  Bertram F. Malle,et al.  An Empirical Assessment of Meta-Analytic Practice , 2009 .

[17]  Betsy Jane Becker,et al.  Modeling multivariate effect sizes. , 1988 .

[18]  E. Chambers.  An Introduction to Meta-Analysis With Articles From The Journal of Educational Research (1992-2002) , 2004 .

[19]  J. Sánchez-Meca,et al.  Averaging Dependent Effect Sizes in Meta-Analysis: a Cautionary Note about Procedures , 1999, The Spanish Journal of Psychology.

[20]  Ingram Olkin,et al.  Stochastically dependent effect sizes. , 1994 .

[21]  L. Hedges.  Distribution Theory for Glass's Estimator of Effect size and Related Estimators , 1981 .

[22]  Apa Publications,et al.  Reporting standards for research in psychology: why do we need them? What might they be? , 2008, The American psychologist.

[23]  Steven D. Brown,et al.  Handbook of applied multivariate statistics and mathematical modeling , 2000 .

[24]  Elizabeth Tipton,et al.  Mindfulness Based Stress Reduction (MBSR) for Improving Health, Quality of Life, and Social Functioning in Adults , 2012 .

[25]  Sandra Jo Wilson,et al.  The comparative effectiveness of outpatient treatment for adolescent substance abuse: a meta-analysis. , 2013, Journal of substance abuse treatment.

[26]  Elizabeth Tipton,et al.  Robust variance estimation with dependent effect sizes: practical considerations including a software tutorial in Stata and spss , 2014, Research synthesis methods.

[27]  Mark W. Lipsey,et al.  Dropout Prevention and Intervention Programs: Effects on School Completion and Dropout Among School-Aged Children and Youth: A Systematic Review , 2010 .

[28]  Wim Van den Noortgate,et al.  Three-level meta-analysis of dependent effect sizes , 2012, Behavior Research Methods.

[29]  S. Graham,et al.  Writing to Read: A Meta-Analysis of the Impact of Writing and Writing Instruction on Reading. , 2011 .

[30]  S. Vaughn,et al.  Interventions for Adolescent Struggling Readers: A Meta-Analysis with Implications for Practice. , 2007 .

[31]  Thomas E. Scruggs,et al.  Reading Comprehension Instruction for Students With Learning Disabilities, 1995—2006: A Meta-Analysis , 2010 .

[32]  S. Vaughn,et al.  A Synthesis of Reading Interventions and Effects on Reading Comprehension Outcomes for Older Struggling Readers , 2009, Review of educational research.

[33]  Nicholas D. Myers,et al.  A Review of Meta-Analyses in Education , 2012 .

[34]  L. Hedges,et al.  Multiple Comparisons within a Study , 2009, Introduction to Meta‐Analysis.

[35]  S. Raudenbush,et al.  A multivariate mixed linear model for meta-analysis. , 1996 .

[36]  H. Swanson,et al.  Instructing Struggling Older Readers: A Selective Meta-Analysis of Intervention Research , 2012 .