Optimized variance estimation under interference and complex experimental designs

Unbiased and consistent variance estimators generally do not exist for design-based treatment effect estimators because experimenters never observe more than one potential outcome for any unit. The problem is exacerbated by interference and complex experimental designs. In this paper, we consider variance estimation for linear treatment effect estimators under interference and arbitrary experimental designs. Experimenters must accept conservative estimators in this setting, but they can strive to minimize the conservativeness. We show that this task can be interpreted as an optimization problem in which one aims to find the lowest estimable upper bound of the true variance given one’s risk preference and knowledge of the potential outcomes. We characterize the set of admissible bounds in the class of quadratic forms, and we demonstrate that the optimization problem is a convex program for many natural objectives. This allows experimenters to construct less conservative variance estimators, making inferences about treatment effects more informative. The resulting estimators are guaranteed to be conservative regardless of whether the background knowledge used to construct the bound is correct, but the estimators are less conservative if the knowledge is reasonably accurate.

[1]  M. B. Wilk,et al.  THE RANDOMZATION ANALYSIS OF A GENERALIZED RANDOMIZED BLOCK DESIGN , 1955 .

[2]  Donald K. K. Lee,et al.  SHARP BOUNDS ON THE VARIANCE IN RANDOMIZED EXPERIMENTS , 2014, 1405.6555.

[3]  Ruodu Wang,et al.  The directional optimal transport , 2020, The Annals of Applied Probability.

[4]  Oscar Kempthorne,et al.  THE RANDOMIZATION THEORY OF' EXPERIMENTAL INFERENCE* , 1955 .

[5]  R. Tyrrell Rockafellar,et al.  Lagrange Multipliers and Optimality , 1993, SIAM Rev..

[6]  Peter M. Aronow,et al.  On equivalencies between design-based and regression-based variance estimators for randomized experiments , 2012 .

[7]  Peter M. Aronow,et al.  Conservative variance estimation for sampling designs with zero pairwise inclusion probabilities , 2013 .

[8]  J. Sekhon,et al.  Blocking estimators and inference under the Neyman-Rubin model , 2015, 1510.01103.

[9]  Stefan Wager,et al.  High-dimensional regression adjustments in randomized experiments , 2016, Proceedings of the National Academy of Sciences.

[10]  Lorenzo Fattorini,et al.  Applying the Horvitz-Thompson criterion in complex designs: A computer-intensive perspective for estimating inclusion probabilities , 2006 .

[11]  Kosuke Imai,et al.  (www.interscience.wiley.com) DOI: 10.1002/sim.3337 Variance identification and efficiency analysis in randomized experiments under the matched-pair design , 2022 .

[12]  Joel A. Middleton,et al.  A Unified Theory of Regression Adjustment for Design-based Inference , 2018, 1803.06011.

[13]  D. Horvitz,et al.  A Generalization of Sampling Without Replacement from a Finite Universe , 1952 .

[14]  Fredrik Savje Causal inference with misspecified exposure mappings , 2021 .

[15]  Colin B. Fogarty On mitigating the analytical limitations of finely stratified experiments , 2017, Journal of the Royal Statistical Society: Series B (Statistical Methodology).

[16]  Joel A. Middleton,et al.  A Class of Unbiased Estimators of the Average Treatment Effect in Randomized Experiments , 2013 .

[17]  Nicole E. Pashley,et al.  Insights on Variance Estimation for Blocked and Matched Pairs Designs , 2017, Journal of Educational and Behavioral Statistics.

[18]  M. Hudgens,et al.  Toward Causal Inference With Interference , 2008, Journal of the American Statistical Association.

[19]  J. Robins,et al.  Double/Debiased Machine Learning for Treatment and Structural Parameters , 2017 .

[20]  J M Robins,et al.  Confidence intervals for causal parameters. , 1988, Statistics in medicine.

[21]  C. Cassel,et al.  Some results on generalized difference estimation and generalized regression estimation for finite populations , 1976 .

[22]  Joel A. Middleton Unifying Design-based Inference: A New Variance Estimation Principle , 2021 .

[23]  W. Lin,et al.  Agnostic notes on regression adjustments to experimental data: Reexamining Freedman's critique , 2012, 1208.2301.

[24]  E. Airoldi,et al.  A systematic investigation of classical causal inference strategies under mis-specification due to network interference , 2018, 1810.08259.

[25]  David A. Freedman,et al.  On regression adjustments to experimental data , 2008, Adv. Appl. Math..

[26]  Peter M. Aronow,et al.  Estimating Average Causal Effects Under General Interference , 2012 .

[27]  Guido W. Imbens,et al.  Estimation of the Conditional Variance in Paired Experiments , 2008 .

[28]  D. Rubin,et al.  Causal Inference for Statistics, Social, and Biomedical Sciences: Sensitivity Analysis and Bounds , 2015 .

[29]  Edoardo M. Airoldi,et al.  Model-assisted design of experiments in the presence of network-correlated outcomes , 2015, Biometrika.

[30]  D. Basu,et al.  An Essay on the Logical Foundations of Survey Sampling, Part One* , 2011 .

[31]  Johann A. Gagnon-Bartsch,et al.  The LOOP Estimator: Adjusting for Covariates in Randomized Experiments , 2017, Evaluation review.

[32]  Carl-Erik Särndal,et al.  Model Assisted Survey Sampling , 1997 .

[33]  G. Gadbury Randomization Inference and Bias of Standard Errors , 2001 .

[34]  D. Rubin [On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9.] Comment: Neyman (1923) and Causal Inference in Experiments and Observational Studies , 1990 .

[35]  Stefan Wager,et al.  Random Graph Asymptotics for Treatment Effect Estimation under Network Interference , 2020 .

[36]  P. Holland Statistics and Causal Inference , 1985 .

[37]  G. Imbens,et al.  A Causal Bootstrap , 2018, The Annals of Statistics.

[38]  Iain Dunning,et al.  JuMP: A Modeling Language for Mathematical Optimization , 2015, SIAM Rev..

[39]  D. Rubin,et al.  Using Standard Tools From Finite Population Sampling to Improve Causal Inference for Complex Experiments , 2018 .