Covariate Balancing Sensitivity Analysis for Extrapolating Randomized Trials across Locations

The ability to generalize experimental results from randomized control trials (RCTs) across locations is crucial for informing policy decisions in targeted regions. Such generalization is often hindered by the lack of identifiability due to unmeasured effect modifiers that compromise direct transport of treatment effect estimates from one location to another. We build upon sensitivity analysis in observational studies and propose an optimization procedure that allows us to get bounds on the treatment effects in targeted regions. Furthermore, we construct more informative bounds by balancing on the moments of covariates. In simulation experiments, we show that the covariate balancing approach is promising in getting sharper identification intervals.

[1]  Rajeev Dehejia,et al.  From Local to Global: External Validity in a Fertility Natural Experiment , 2015, Journal of Business & Economic Statistics.

[2]  Karsten M. Borgwardt,et al.  Covariate Shift by Kernel Mean Matching , 2009, NIPS 2009.

[3]  V. J. Hotz,et al.  Predicting the efficacy of future training programs using past experiences at other locations , 2005 .

[4]  Dylan S. Small,et al.  Sensitivity Analysis for Multiple Comparisons in Matched Observational Studies Through Quadratically Constrained Linear Programming , 2015, 1505.03810.

[5]  Eloise E Kaizar.  Estimating treatment effect via simple cross design synthesis , 2011, Statistics in medicine.

[6]  Takafumi Kanamori,et al.  Density Ratio Estimation in Machine Learning , 2012 .

[7]  S. Cole,et al.  Correction: Sensitivity analysis for an unobserved moderator in RCT-to-target-population generalization of treatment effects , 2016, The Annals of Applied Statistics.

[8]  Hamsa Bastani.  Predicting with Proxies: Transfer Learning in High Dimension , 2018 .

[9]  Yang Ning,et al.  High Dimensional Propensity Score Estimation via Covariate Balancing , 2017 .

[10]  G. Imbens,et al.  Estimating Treatment Effects using Multiple Surrogates: The Role of the Surrogate Score and the Surrogate Index , 2016, 1603.09326.

[11]  Elias Bareinboim,et al.  Meta-Transportability of Causal Effects: A Formal Approach , 2013, AISTATS.

[12]  Arian Maleki,et al.  Minimax Linear Estimation of the Retargeted Mean , 2019, 1901.10296.

[13]  John Duchi,et al.  Bounds on the conditional and average treatment effect in the presence of unobserved confounders , 2018 .

[14]  Zhiqiang Tan,et al.  Model-assisted inference for treatment effects using regularized calibrated estimation with high-dimensional data , 2018, The Annals of Statistics.

[15]  K. Imai,et al.  Covariate balancing propensity score , 2014 .

[16]  S. Cole,et al.  Generalizing evidence from randomized clinical trials to target populations: The ACTG 320 trial. , 2010, American journal of epidemiology.

[17]  E. Oster,et al.  Weighting for External Validity , 2017 .

[18]  P. Rosenbaum.  Weighted M-statistics With Superior Design Sensitivity in Matched Observational Studies With Multiple Controls , 2014 .

[19]  Stephen H Bell,et al.  External Validity in Policy Evaluations that Choose Sites Purposively. , 2013, Journal of policy analysis and management : [the journal of the Association for Public Policy Analysis and Management].

[20]  Qingyuan Zhao.  Covariate balancing propensity score by tailored loss functions , 2016, The Annals of Statistics.

[21]  G. W. Imbens.  Sensitivity to Exogeneity Assumptions in Program Evaluation , 2003 .

[22]  Bryan S. Graham,et al.  Efficient Estimation of Data Combination Models by the Method of Auxiliary-to-Study Tilting (AST) , 2011 .

[23]  C. Shen,et al.  Sensitivity analysis for causal inference using inverse probability weighting , 2011, Biometrical journal. Biometrische Zeitschrift.

[24]  Donald K. K. Lee,et al.  Interval estimation of population means under unknown but bounded probabilities of sample selection , 2013 .

[25]  Stephen R Cole,et al.  Transportability of Trial Results Using Inverse Odds of Sampling Weights. , 2017, American journal of epidemiology.

[26]  Stefan Wager,et al.  Shape-constrained partial identification of a population mean under unknown probabilities of sample selection , 2017, 1706.07550.

[27]  Jens Hainmueller,et al.  Entropy Balancing for Causal Effects: A Multivariate Reweighting Method to Produce Balanced Samples in Observational Studies , 2012, Political Analysis.

[28]  Nathan Kallus,et al.  Balanced Policy Evaluation and Learning , 2017, NeurIPS.

[29]  P. Rosenbaum.  Sensitivity analysis for certain permutation inferences in matched observational studies , 1987 .

[30]  E. C. Hammond,et al.  Smoking and lung cancer: recent evidence and a discussion of some questions. , 1959, Journal of the National Cancer Institute.

[31]  Tyler J. VanderWeele,et al.  Sensitivity Analysis in Observational Research: Introducing the E-Value , 2017, Annals of Internal Medicine.

[32]  Costas Meghir,et al.  Subsidizing Vocational Training for Disadvantaged Youth in Colombia: Evidence from a Randomized Trial , 2011 .

[33]  Juan Pablo Vielma,et al.  Building Representative Matched Samples With Multi-Valued Treatments in Large Observational Studies , 2018, Journal of Computational and Graphical Statistics.

[34]  T. Shakespeare,et al.  Observational Studies , 2003 .

[35]  José R. Zubizarreta,et al.  Minimal Approximately Balancing Weights: Asymptotic Properties and Practical Considerations , 2017 .

[36]  S. Kruger.  Design Of Observational Studies , 2016 .

[37]  Dylan S. Small,et al.  Sensitivity analysis for inverse probability weighting estimators via the percentile bootstrap , 2017, Journal of the Royal Statistical Society: Series B (Statistical Methodology).

[38]  Uri Shalit,et al.  Removing Hidden Confounding by Experimental Grounding , 2018, NeurIPS.

[39]  D. Rubin,et al.  The central role of the propensity score in observational studies for causal effects , 1983 .

[40]  Stefan Wager,et al.  Sparsity Double Robust Inference of Average Treatment Effects , 2019, 1905.00744.

[41]  Elias Bareinboim,et al.  External Validity: From Do-Calculus to Transportability Across Populations , 2014, Probabilistic and Causal Inference.

[42]  Elias Bareinboim,et al.  Causal inference and the data-fusion problem , 2016, Proceedings of the National Academy of Sciences.

[43]  Zhiqiang Tan,et al.  A Distributional Approach for Causal Inference Using Propensity Scores , 2006 .

[44]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[45]  Paul R Rosenbaum,et al.  A New u‐Statistic with Superior Design Sensitivity in Matched Observational Studies , 2011, Biometrics.

[46]  Eloise E Kaizar,et al.  The use of propensity scores and observational data to estimate randomized controlled trial generalizability bias , 2013, Statistics in medicine.

[47]  Stefan Wager,et al.  Augmented minimax linear estimation , 2017, The Annals of Statistics.

[48]  G. Imbens,et al.  Approximate residual balancing: debiased inference of average treatment effects in high dimensions , 2016, 1604.07125.

[49]  Evan T. R. Rosenman,et al.  Propensity score methods for merging observational and experimental datasets , 2018, Statistics in medicine.

[50]  Zhiqiang Tan,et al.  Regularized calibrated estimation of propensity scores with model misspecification and high-dimensional data , 2017, Biometrika.

[51]  Elias Bareinboim,et al.  Transportability from Multiple Environments with Limited Experiments: Completeness Results , 2014, NIPS.

[52]  J. Zubizarreta.  Stable Weights that Balance Covariates for Estimation With Incomplete Outcome Data , 2015 .

[53]  Jacob Alex Klerman,et al.  NBER WORKING PAPER SERIES EVALUATING THE DIFFERENTIAL EFFECTS OF ALTERNATIVE WELFARE-TO-WORK TRAINING COMPONENTS: A RE-ANALYSIS OF THE CALIFORNIA GAIN PROGRAM , 2022 .

[54]  Y. Qin.  Inferences for case-control and semiparametric two-sample density ratio models , 1998 .

[55]  Rachael Meager.  Aggregating Distributional Treatment Effects: A Bayesian Hierarchical Analysis of the Microcredit Literature , 2017, American Economic Review.

[56]  Paul R Rosenbaum,et al.  Attributing Effects to Treatment in Matched Observational Studies , 2002 .

[57]  Catherine P. Bradshaw,et al.  The use of propensity scores to assess the generalizability of results from randomized trials , 2011, Journal of the Royal Statistical Society. Series A,.