Covariance Adjustments for the Analysis of Randomized Field Experiments

Background: It has become common practice to analyze randomized experiments using linear regression with covariates. Improved precision of treatment effect estimates is the usual motivation. In a series of important articles, David Freedman showed that this approach can be badly flawed. Recent work by Winston Lin offers partial remedies, but important problems remain. Results: In this article, we address those problems through a reformulation of the Neyman causal model. We provide a practical estimator and valid standard errors for the average treatment effect. Proper generalizations to well-defined populations can follow. Conclusion: In most applications, the use of covariates to improve precision is not worth the trouble.

[1]  David R. Cox.  Planning of Experiments , 1958 .

[2]  P. Holland.  Statistics and Causal Inference , 1985 .

[3]  J. Rice.  Mathematical Statistics and Data Analysis , 1988 .

[4]  D. Rubin.  [On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9.] Comment: Neyman (1923) and Causal Inference in Experiments and Observational Studies , 1990 .

[5]  James J. Heckman,et al.  Assessing the Case for Social Experiments , 1995 .

[6]  E. Lehmann.  Elements of large-sample theory , 1998 .

[7]  Charles S. Reichardt,et al.  Justifying the use and increasing the power of a t test for a randomized experiment with a convenience sample. , 1999 .

[8]  Richard A. Berk,et al.  Statistical Assumptions as Empirical Commitments , 2001 .

[9]  H. Leeb,et al.  CAN ONE ESTIMATE THE UNCONDITIONAL DISTRIBUTION OF POST-MODEL-SELECTION ESTIMATORS? , 2003, Econometric Theory.

[10]  R. Berk.  Regression Analysis: A Constructive Critique , 2003 .

[11]  Richard A. Berk,et al.  Randomized experiments as the bronze standard , 2005 .

[12]  D. Freedman.  Statistical Models for Causation , 2006, Evaluation review.

[13]  Richard A. Berk,et al.  Statistical inference and meta-analysis , 2007 .

[14]  Amery D. Wu,et al.  Understanding and Using Mediators and Moderators , 2008 .

[15]  D. Freedman.  Statistical Models and Causal Inference: On Regression Adjustments in Experiments with Several Treatments , 2008, 0803.3757.

[16]  David A. Freedman,et al.  On regression adjustments to experimental data , 2008, Adv. Appl. Math..

[17]  P. Rosenbaum.  Design of Observational Studies , 2009, Springer Series in Statistics.

[18]  R. Berk,et al.  When second best is good enough: a comparison between a true experiment and a regression discontinuity quasi-experiment , 2010 .

[19]  Bruce G. Taylor,et al.  The impact of length of domestic violence treatment on the patterns of subsequent intimate partner violence , 2010 .

[20]  Michele Staton‐Tindall,et al.  A multi-site, randomized study of strengths-based case management with substance-abusing parolees , 2011, Journal of experimental criminology.

[21]  W. Lin,et al.  Agnostic notes on regression adjustments to experimental data: Reexamining Freedman's critique , 2012, 1208.2301.

[22]  N. Hipple,et al.  Long-term impact of family group conferences on re-offending: the Indianapolis restorative justice experiment , 2012 .

[23]  Bruce G. Taylor,et al.  A randomized test of initial and residual deterrence from directed patrols and use of license plate readers at crime hot spots , 2013 .

[24]  Luke W. Miratrix,et al.  Adjusting treatment effect estimates by post‐stratification in randomized experiments , 2013 .

[25]  B. Vila,et al.  Results from experimental trials testing participant responses to White, Hispanic and Black suspects in high-fidelity deadly force judgment and decision-making simulations , 2013 .

[26]  M. Deković,et al.  A randomized controlled trial of the effectiveness of multisystemic therapy in the Netherlands: post-treatment changes and moderator effects , 2013 .

[27]  Andreas Buja,et al.  Misspecified Mean Function Regression , 2014 .

[28]  Dennis P. Rosenbaum,et al.  Building group capacity for problem solving and police–community partnerships through survey feedback and training: a randomized control trial within Chicago’s community policing program , 2014 .

[29]  Winston T. Lin.  Comments on “Covariance Adjustments for the Analysis of Randomized Field Experiments” , 2014, Evaluation review.