Going beyond simple sample size calculations: a practitioner's guide

Basic methods to compute required sample sizes are well understood and supported by widely available software. However, the sophistication of the methods commonly used has not kept pace with the complexity of commonly employed experimental designs. We compile available methods for sample size calculations for continuous and binary outcomes with and without covariates, for both clustered and non-clustered RCTs. Formulae for both panel data and unbalanced designs are provided. Extensions include methods to: (1) optimise the sample when costs constraints are binding, (2) compute the power of a complex design by simulation, and (3) adjust calculations for multiple testing. Click here to view accompanying sample size calculators for this paper.

[1]  N. Kumagai,et al.  Simulation Program to Determine Sample Size and Power for a Multiple Logistic Regression Model with Unspecified Covariate Distributions , 2014 .

[2]  王林,et al.  CONSORT , 2011 .

[3]  Eugene Demidenko,et al.  Sample size determination for logistic regression revisited , 2006, Statistics in medicine.

[4]  C. Lloyd.  Estimating test power adjusted for size , 2005 .

[5]  Larry V. Hedges,et al.  Statistical Power Analysis in Education Research , 2010 .

[6]  D. Moher,et al.  CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials , 2010, Journal of pharmacology & pharmacotherapeutics.

[7]  James J. Heckman,et al.  Assessing the Case for Social Experiments , 1995 .

[8]  Cora J. M. Maas,et al.  Optimal Experimental Designs for Multilevel Logistic Models with Two Binary Predictors , 2005 .

[9]  Richard Hooper,et al.  Versatile Sample-Size Calculation using Simulation , 2013 .

[10]  J. Wason,et al.  Correcting for multiple-testing in multi-arm trials: is it necessary and is it done? , 2014, Trials.

[11]  A. H. Feiveson,et al.  Power by Simulation , 2002 .

[12]  S. S. Young,et al.  Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment , 1993 .

[13]  H. Watanabe,et al.  Points to Consider on Multiplicity Issues in Clinical Trials , 2006 .

[14]  Peter Z. Schochet.  Statistical Power for School-Based RCTs With Binary Outcomes , 2013 .

[15]  H. Bloom,et al.  Using Covariates to Improve Precision for Studies That Randomize Schools to Evaluate Educational Interventions , 2007 .

[16]  Steven Teerenstra,et al.  A simple sample size formula for analysis of covariance in cluster randomized trials , 2012, Statistics in medicine.

[17]  P. Carneiro,et al.  Long Term Impacts of Compensatory Preschool on Health and Behavior: Evidence from Head Start , 2012, SSRN Electronic Journal.

[18]  Xiaofeng Steven Liu,et al.  Statistical Power Analysis for the Social and Behavioral Sciences: Basic and Advanced Techniques , 2013 .

[19]  Esther Duflo,et al.  Advances in Economics and Econometrics: Field Experiments in Development Economics , 2006 .

[20]  David J. McKenzie,et al.  Beyond Baseline and Follow-Up: The Case for More T in Experiments , 2011 .

[21]  Michael Kremer,et al.  Chapter 61 Using Randomization in Development Economics Research: A Toolkit ★ , 2007 .

[22]  K. Young.  4. The Collected Works of John W. Tukey: Vol. VIII, Multiple Comparisons: 1948 , 1995 .

[23]  Steven D. Levitt,et al.  FIELD EXPERIMENTS IN ECONOMICS : THE PAST , THE PRESENT , AND THE FUTURE , 2008 .

[24]  Allan Donner,et al.  Design and Analysis of Cluster Randomization Trials in Health Research , 2001 .

[25]  Ken Kleinman,et al.  Calculating Power by Bootstrap, with an Application to Cluster-Randomized Trials , 2014, EGEMS.

[26]  Larry V. Hedges,et al.  Statistical power analysis , 2010 .

[27]  Evangelos Kontopantelis,et al.  Simulation-Based Power Calculations for Mixed Effects Modeling: ipdpower in Stata , 2016 .

[28]  Monica Costa Dias,et al.  Alternative approaches to evaluation in empirical microeconomics , 2002, The Journal of Human Resources.

[29]  Gary Burtless,et al.  The Case for Randomized Field Trials in Economic and Policy Research , 1995 .

[30]  Henry Braun,et al.  The Collected Works of John W. Tukey, Volume VIII: Multiple Comparisons, 1948-1983 , 1994 .

[31]  森川 敏彦,et al.  COMMITTEE FOR PROPRIETARY MEDICINAL PRODUCTS ( CPMP ) POINTS TO CONSIDER ON MULTIPLICITY ISSUES IN CLINICAL TRIALS , 2002 .

[32]  Michael Wolf,et al.  Centre De Referència En Economia Analítica Barcelona Economics Working Paper Series Working Paper Nº 17 Stewise Multiple Testing as Formalized Data Snooping Stepwise Multiple Testing as Formalized Data Snooping , 2022 .

[33]  Michael L. Anderson.  Multiple Inference and Gender Differences in the Effects of Early Intervention: A Reevaluation of the Abecedarian, Perry Preschool, and Early Training Projects , 2008 .