PowerUp!: A Tool for Calculating Minimum Detectable Effect Sizes and Minimum Required Sample Sizes for Experimental and Quasi-Experimental Design Studies

Abstract This paper and the accompanying tool are intended to complement existing supports for conducting power analysis tools by offering a tool based on the framework of Minimum Detectable Effect Sizes (MDES) formulae that can be used in determining sample size requirements and in estimating minimum detectable effect sizes for a range of individual- and group-random assignment design studies and for common quasi-experimental design studies. The paper and accompanying tool cover computation of minimum detectable effect sizes under the following study designs: individual random assignment designs, hierarchical random assignment designs (2-4 levels), block random assignment designs (2-4 levels), regression discontinuity designs (6 types), and short interrupted time-series designs. In each case, the discussion and accompanying tool consider the key factors associated with statistical power and minimum detectable effect sizes, including the level at which treatment occurs and the statistical models (e.g., fixed effect and random effect) used in the analysis. The tool also includes a module that estimates for one and two level random assignment design studies the minimum sample sizes required in order for studies to attain user-defined minimum detectable effect sizes.

[1]  H. Bloom,et al.  Using Covariates to Improve Precision for Studies That Randomize Schools to Evaluate Educational Interventions , 2007 .

[2]  P. Gleason,et al.  The Evaluation of Charter School Impacts: Final Report. NCEE 2010-4029. , 2010 .

[3]  Spyros Konstantopoulos,et al.  The Power of the Test for Treatment Effects in Three-Level Cluster Randomized Designs , 2008 .

[4]  L. Hedges,et al.  Intraclass Correlation Values for Planning Group-Randomized Trials in Education , 2007 .

[5]  Oshua,et al.  USING MAIMONIDES’ RULE TO ESTIMATE THE EFFECT OF CLASS SIZE ON SCHOLASTIC ACHIEVEMENT* , 2003 .

[6]  J. Angrist,et al.  Using Maimonides&Apos; Rule to Estimate the Effect of Class Size on Student Achievement , 1997 .

[7]  Geoffrey D. Borman,et al.  Final Reading Outcomes of the National Randomized Field Trial of Success for All , 2007 .

[8]  Larry V. Hedges,et al.  Statistical Power Analysis in Education Research , 2010 .

[9]  Thomas J. Smith,et al.  Making Progress Toward Graduation Evidence from the Talent Development High School Model , 2005 .

[10]  H. Bloom.  Estimating Program Impacts on Student Achievement Using “ Short ” Interrupted Time Series , 1999 .

[11]  H. Bloom.  Measuring the Impacts of Whole-School Reforms: Methodological Lessons from an Evaluation of Accelerated Schools. MDRC Working Papers on Research Methodology. , 2001 .

[12]  S. Raudenbush.  Statistical analysis and optimal design for cluster randomized trials , 1997 .

[13]  Paul Montgomery,et al.  Experimental and Quasi-Experimental Designs , 2011 .

[14]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[15]  John Deke,et al.  Precision Gains from Publically Available School Proficiency Measures Compared to StudyCollected Test Scores in Education ClusterRandomized Trials , 2010 .

[16]  Theresa M. Akey,et al.  The Challenge of Scaling Up Educational Reform. Findings and Lessons from First Things First. Final Report. , 2005 .

[17]  Larry V. Hedges,et al.  The Long-Term Effects of Small Classes: A Five-Year Follow-Up of the Tennessee Class Size Experiment , 1999 .

[18]  W. C. Smith,et al.  Reading First Impact Study: Interim Report , 2008 .

[19]  Using Student Data to Improve Teaching and Learning: Findings from an Evaluation of the Formative Assessments of Students Thinking in Reading (FAST-R) Program in Boston Elementary Schools. , 2008 .

[20]  P. Gleason,et al.  The Evaluation of Charter School Impacts , 2010 .

[21]  S. Raudenbush,et al.  Strategies for Improving Precision in Group-Randomized Experiments , 2007 .

[22]  William M. K. Trochim,et al.  Research Design for Program Evaluation: The Regression-Discontinuity Approach , 1984 .

[23]  H. Bloom,et al.  Using “Short” Interrupted Time-Series Analysis To Measure The Impacts Of Whole-School Reforms , 2003, Evaluation review.

[24]  Larry L. Orr,et al.  Social Experiments: Evaluating Public Programs With Experimental Methods , 1998 .

[25]  R. Murnane,et al.  Methods Matter: Improving Causal Inference in Educational and Social Science Research , 2010 .

[26]  S. Raudenbush,et al.  Statistical power and optimal design for multisite randomized trials. , 2000, Psychological methods.

[27]  D. Campbell,et al.  Regression-Discontinuity Analysis: An Alternative to the Ex-Post Facto Experiment , 1960 .

[28]  H. Bloom.  Learning more from social experiments: evolving analytic approaches , 2006 .

[29]  Jordan D. Matsudaira.  Mandatory summer school and student achievement , 2008 .

[30]  Spyros Konstantopoulos,et al.  Using Power Tables to Compute Statistical Power in Multilevel Experimental Designs , 2009 .

[31]  M. Lipsey,et al.  Performance Trajectories and Performance Gaps as Achievement Effect-Size Benchmarks for Educational Interventions , 2008 .

[32]  Robert F. Boruch,et al.  Randomized Experiments for Planning and Evaluation: A Practical Guide , 1998 .

[33]  David M. Murray,et al.  Design and Analysis of Group- Randomized Trials , 1998 .

[34]  Peter Z. Schochet.  Technical Methods Report: Statistical Power for Regression Discontinuity Designs in Education Evaluations. NCEE 2008-4026. , 2008 .

[35]  Howard S. Bloom,et al.  The Core Analytics of Randomized Experiments for Social Research. MDRC Working Papers on Research Methodology. , 2006 .

[36]  Howard S. Bloom,et al.  Minimum Detectable Effects , 1995 .

[37]  Thomas D. Cook,et al.  Comer's School Development Program in Chicago: A Theory-Based Evaluation , 2000 .

[38]  Peter Z. Schochet.  Statistical Power for Random Assignment Evaluations of Education Programs , 2005 .

[39]  Larry V. Hedges,et al.  Statistical Power Analysis in Education Research. NCSER 2010-3006. , 2010 .

[40]  R. Kirk.  Experimental Design: Procedures for the Behavioral Sciences , 1970 .