Treatment effect estimation with Multilevel Regression and Poststratification

Multilevel regression and poststratification (MRP) is a flexible modeling technique that has been used in a broad range of small-area estimation problems. Traditionally, MRP studies have been focused on non-causal settings, where estimating a single population value using a nonrepresentative sample was of primary interest. In this manuscript, MRP-style estimators will be evaluated in an experimental causal inference setting. We simulate a large-scale randomized control trial with a stratified cluster sampling design, and compare traditional and nonparametric treatment effect estimation methods with MRP methodology. Using MRP-style estimators, treatment effect estimates for areas as small as 1.3% of the population have lower bias and variance than standard causal inference methods, even in the presence of treatment effect heterogeneity. The design of our simulation studies also requires us to build upon a MRP variant that allows for non-census covariates to be incorporated into poststratification.

[1]  Daniel Simpson,et al.  Improving multilevel regression and poststratification with structured priors. , 2019, Bayesian analysis.

[2]  Paul-Christian Bürkner,et al.  brms: An R Package for Bayesian Multilevel Models Using Stan , 2017 .

[3]  Jennifer Hill,et al.  Automated versus Do-It-Yourself Methods for Causal Inference: Lessons Learned from a Data Analysis Competition , 2017, Statistical Science.

[4]  BARP: Improving Mister P Using Bayesian Additive Regression Trees , 2019, American Political Science Review.

[5]  R. Little.  Post-Stratification: A Modeler's Perspective , 1993 .

[6]  Jiqiang Guo,et al.  Stan: A Probabilistic Programming Language. , 2017, Journal of statistical software.

[7]  D. Rubin.  Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[8]  Joseph Kang,et al.  Demystifying Double Robustness: A Comparison of Alternative Strategies for Estimating a Population Mean from Incomplete Data , 2007, 0804.2958.

[9]  David M. Rothschild,et al.  Forecasting elections with non-representative polls , 2015 .

[10]  E. Stuart,et al.  Implementing statistical methods for generalizing randomized trial findings to a target population. , 2019, Addictive behaviors.

[11]  Andrew Gelman,et al.  Bayesian Multilevel Estimation with Poststratification: State-Level Estimates from National Polls , 2004, Political Analysis.

[12]  Elizabeth A. Stuart,et al.  Theory and practice in non-probability surveys : parallels between causal inference and survey inference , 2017 .

[13]  Susan Athey,et al.  The Econometrics of Randomized Experiments , 2016, 1607.00698.

[14]  Paul-Christian Bürkner,et al.  Advanced Bayesian Multilevel Modeling with the R Package brms , 2017, R J..

[15]  D. V. Lindley,et al.  Randomization Analysis of Experimental Data: The Fisher Randomization Test Comment , 1980 .

[16]  Jennifer Hill,et al.  Assessing Methods for Generalizing Experimental Impact Estimates to Target Populations , 2016, Journal of research on educational effectiveness.

[17]  H. Chipman,et al.  BART: Bayesian Additive Regression Trees , 2008, 0806.3286.

[18]  Catherine P. Bradshaw,et al.  The use of propensity scores to assess the generalizability of results from randomized trials , 2011, Journal of the Royal Statistical Society. Series A,.

[19]  Megan S. Schuler,et al.  Generalizing observational study results: applying propensity score methods to complex surveys. , 2014, Health services research.

[20]  Chris S. Hulleman,et al.  A national experiment reveals where a growth mindset improves achievement , 2019, Nature.

[21]  Elizabeth A Stuart,et al.  Generalizability of Randomized Trial Results to Target Populations , 2018, Research on social work practice.

[22]  John K Kruschke,et al.  Bayesian data analysis. , 2010, Wiley interdisciplinary reviews. Cognitive science.

[23]  Mark J. van der Laan,et al.  Targeted Maximum Likelihood Estimation: A Gentle Introduction , 2009 .

[24]  Benjamin E. Lauderdale,et al.  Model-based pre-election polling for national and sub-national outcomes in the US and UK , 2020, International Journal of Forecasting.

[25]  Thomas Lumley,et al.  Fitting Regression Models to Survey Data , 2017 .

[26]  Robert D. Tortora,et al.  Sampling: Design and Analysis , 2000 .

[27]  P. Richard Hahn,et al.  Bayesian Regression Tree Models for Causal Inference: Regularization, Confounding, and Heterogeneous Effects , 2017, 1706.09523.

[28]  D. Green,et al.  Modeling Heterogeneous Treatment Effects in Survey Experiments with Bayesian Additive Regression Trees , 2012 .

[29]  Hadley Wickham,et al.  ggplot2 - Elegant Graphics for Data Analysis (2nd Edition) , 2017 .

[30]  Sören R. Künzel,et al.  Metalearners for estimating heterogeneous treatment effects using machine learning , 2017, Proceedings of the National Academy of Sciences.

[31]  Juned Siddique,et al.  Generalizing randomized trial findings to a target population using complex survey population data , 2020, Statistics in medicine.

[32]  A. Coppock,et al.  Declaring and Diagnosing Research Designs , 2019, American Political Science Review.

[33]  A. Raftery,et al.  Strictly Proper Scoring Rules, Prediction, and Estimation , 2007 .

[34]  Luke W. Miratrix,et al.  Worth Weighting ? How to Think About and Use Sample Weights in Survey Experiments , 2017 .

[35]  Andrew Gelman,et al.  Know your population and know your model: Using model-based regression and poststratification to generalize findings beyond the observed sample. , 2019, Psychological methods.

[36]  S. Athey,et al.  Estimating Treatment Effects with Causal Forests: An Application , 2019, Observational Studies.

[37]  Joseph Hilbe,et al.  Data Analysis Using Regression and Multilevel/Hierarchical Models , 2009 .

[38]  Jennifer L. Hill,et al.  Bayesian Nonparametric Modeling for Causal Inference , 2011 .

[39]  Luke W. Miratrix,et al.  Bridging Finite and Super Population Causal Inference , 2017, 1702.08615.

[40]  Jeffrey R. Lax,et al.  The Party or the Purse? Unequal Representation in the US Senate , 2019, American Political Science Review.

[41]  Stefan Wager,et al.  Estimation and Inference of Heterogeneous Treatment Effects using Random Forests , 2015, Journal of the American Statistical Association.

[42]  Aki Vehtari,et al.  Visualization in Bayesian workflow , 2017, Journal of the Royal Statistical Society: Series A (Statistics in Society).

[43]  W. Deming,et al.  On a Least Squares Adjustment of a Sampled Frequency Table When the Expected Marginal Totals are Known , 1940 .

[44]  Jonathan P. Kastellec,et al.  Polarizing the Electoral Connection: Partisan Representation in Supreme Court Confirmation Politics , 2015, The Journal of Politics.

[45]  J. Pearl.  Causal diagrams for empirical research , 1995 .

[46]  Jeffrey R. Lax,et al.  How Should We Estimate Public Opinion in the States , 2009 .

[47]  Elizabeth Tipton,et al.  How Generalizable Is Your Experiment? An Index for Comparing Experimental Samples and Populations , 2014 .