The Perils of Randomization Checks in the Analysis of Experiments

In the analysis of experimental data, randomization checks, also known as balance tests, are used to indicate whether a randomization has produced balance on various characteristics across experimental conditions. Randomization checks are popular in many fields although their merits have yet to be established. The grounds on which balance tests are generally justified include either 1) the credibility of experimental findings, and/or 2) the efficiency of the statistical model. We show that balance tests cannot improve either credibility or efficiency. The most common “remedy” resulting from a failed balance test is the inclusion as a covariate of a variable failing the test; this practice cannot improve the choice of statistical model. Other commonly suggested responses to failed balance tests such as post-stratification or re-randomization also fail to improve on methods that do not require balance tests. We advocate resisting reviewer requests for randomization checks in all but some narrowly defined circumstances.

[1]  R. Abelson.  Statistics As Principled Argument , 1995 .

[2]  Diana C. Mutz.  Population-Based Survey Experiments , 2011 .

[3]  R. Little.  Post-Stratification: A Modeler's Perspective , 1993 .

[4]  Gary King,et al.  Misunderstandings between experimentalists and observationalists about causal inference , 2008 .

[5]  Jake Bowers.  Cambridge Handbook of Experimental Political Science: Making Effects Manifest in Randomized Experiments , 2011 .

[6]  Kosuke Imai,et al.  Do Get-Out-the-Vote Calls Reduce Turnout? The Importance of Statistical Methods for Field Experiments , 2005, American Political Science Review.

[7]  Jake Bowers,et al.  Covariate balance in simple stratified and clustered comparative studies , 2008, 0808.3857.

[8]  D. Altman.  Comparability of Randomised Groups , 1985 .

[9]  S. Pocock,et al.  Subgroup analysis, covariate adjustment and baseline comparisons in clinical trial reporting: current practiceand problems , 2002, Statistics in medicine.

[10]  D. Freedman.  Statistical Models and Causal Inference: On Regression Adjustments in Experiments with Several Treatments , 2008, 0803.3757.

[11]  M. Feldstein.  Multicollinearity and the Mean Square Error of Alternative Estimators , 1973 .

[12]  S. Senn.  Testing for baseline balance in clinical trials. , 1994, Statistics in medicine.

[13]  Richard A. Berk,et al.  Statistical Inference After Model Selection , 2010 .

[14]  C B Begg,et al.  Suspended judgment. Significance tests of covariate imbalance in clinical trials. , 1990, Controlled clinical trials.

[15]  T. Permutt.  Testing for imbalance of covariates in controlled experiments. , 1990, Statistics in medicine.

[16]  M. Davidian,et al.  Covariate adjustment for two‐sample treatment comparisons in randomized clinical trials: A principled yet flexible approach , 2008, Statistics in medicine.

[17]  Kari Lock Morgan.  Rerandomization to Improve Covariate Balance in Randomized Experiments , 2011 .

[18]  D. Green,et al.  The Effects of Canvassing, Telephone Calls, and Direct Mail on Voter Turnout: A Field Experiment , 2000, American Political Science Review.