The Relationship Between Sample Sizes and Effect Sizes in Systematic Reviews in Education

Research in fields other than education has found that studies with small sample sizes tend to have larger effect sizes than those with large samples. This article examines the relationship between sample size and effect size in education. It analyzes data from 185 studies of elementary and secondary mathematics programs that met the standards of the Best Evidence Encyclopedia. As predicted, there was a significant negative correlation between sample size and effect size. The differences in effect sizes between small and large experiments were much greater than those between randomized and matched experiments. Explanations for the effects of sample size on effect size are discussed.

[1]  Carole Torgerson,et al.  The quality of systematic reviews of effectiveness in literacy learning in English: a ‘tertiary’ review , 2007 .

[2]  J. Torgesen,et al.  The Effectiveness of a Group Reading Instruction Program with Poor Readers in Multiple Grades , 2001 .

[3]  Geoffrey D. Borman,et al.  Comprehensive School Reform and Achievement: A Meta-Analysis , 2003 .

[4]  Vivian C. Wong,et al.  Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within‐study comparisons , 2008 .

[5]  Mark W. Lipsey,et al.  Practical Meta-Analysis , 2000 .

[6]  Robert E. Slavin,et al.  Effective Programs in Middle and High School Mathematics: A Best-Evidence Synthesis , 2008 .

[7]  J. Ioannidis,et al.  Issues in comparisons between meta-analyses and large trials. , 1998, JAMA.

[8]  Robert E. Slavin,et al.  Effective Reading Programs for the Elementary Grades: A Best-Evidence Synthesis , 2009 .

[9]  R. Tweedie,et al.  Publication Bias in Meta-Analysis: A Bayesian Data-Augmentation Approach to Account for Issues Exempli(cid:12)ed in the Passive Smoking Debate , 1997 .

[10]  Richard L. Tweedie,et al.  A Non-parametric "Trim and Fill" Method of Assessing Publication Bias in Meta-analysis , 1998 .

[11]  Nancy A. Madden,et al.  Measures Inherent to Treatments in Program Effectiveness Reviews , 2011 .

[12]  J. Sterne,et al.  Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. , 2000, Journal of clinical epidemiology.

[13]  Robert E. Slavin,et al.  Perspectives on Evidence-Based Research in Education—What Works? Issues in Synthesizing Educational Program Evaluations , 2008 .

[14]  T. Conway,et al.  Preventing reading failure in young children with phonological processing disabilities: Group and individual responses to instruction. , 1999 .

[15]  Christian Gluud,et al.  Reported Methodologic Quality and Discrepancies between Large and Small Randomized Trials in Meta-Analyses , 2001, Annals of Internal Medicine.

[16]  Lee J. Cronbach,et al.  Toward Reform of Program Evaluation: Aims, Methods, and Institutional Arrangements. , 1982 .

[17]  Robert E. Slavin,et al.  Effective Programs in Elementary Mathematics: A Best-Evidence Synthesis , 2007 .

[18]  R. Slavin,et al.  Effective Reading Programs for Middle and High Schools: A Best‐Evidence Synthesis , 2008 .

[19]  Wolfgang Viechtbauer,et al.  Publication bias in meta-analysis: Prevention, assessment and adjustments , 2007, Psychometrika.