A Simple Approximation for Evaluating External Validity Bias

We develop a simple approximation that relates the total external validity bias in randomized trials to (i) bias from selection on observables and (ii) a measure for the role of treatment effect heterogeneity in driving selection into the experimental sample.

[1]  Robert B. Olsen,et al.  On the “Where” of Social Experiments: Selecting More Representative Samples to Inform Policy , 2016 .

[2]  Junko Onishi,et al.  Should Aid Reward Performance? Evidence from a Field Experiment on Health and Education in Indonesia , 2012, American economic journal. Applied economics.

[3]  J. Pearl,et al.  Causal inference in statistics , 2016 .

[4]  Roland R. Ramsahai,et al.  From sample average treatment effect to population average treatment effect on the treated: combining experimental with observational studies to estimate population treatment effects , 2015 .

[5]  V. J. Hotz,et al.  Predicting the efficacy of future training programs using past experiences at other locations , 2005 .

[6]  Amanda E. Kowalski.  Doing More When You&Apos;Re Running Late: Applying Marginal Treatment Effect Methods to Examine Treatment Effect Heterogeneity in Experiments , 2016 .

[7]  Catherine P. Bradshaw,et al.  The use of propensity scores to assess the generalizability of results from randomized trials , 2011, Journal of the Royal Statistical Society. Series A,.

[8]  Joshua D. Angrist,et al.  Wanna Get Away? Regression Discontinuity Estimation of Exam School Effects Away From the Cutoff , 2015 .

[9]  Rachel Glennerster,et al.  The Generalizability Puzzle , 2017 .

[10]  A Note on Additive Separability and Latent Index Models of Binary Choice: Representation Results , 2006 .

[11]  J. Heckman.  Sample selection bias as a specification error , 1979 .

[12]  G. Borjas.  Self-Selection and the Earnings of Immigrants , 1987, Foundations of Migration Economics.

[13]  K. Imai,et al.  Covariate balancing propensity score , 2014 .

[14]  J. Angrist,et al.  Identification and Estimation of Local Average Treatment Effects , 1995 .

[15]  Nicholas Bloom,et al.  Does Working from Home Work? Evidence from a Chinese Experiment , 2013 .

[16]  S. Cole,et al.  Correction: Sensitivity analysis for an unobserved moderator in RCT-to-target-population generalization of treatment effects , 2016, The Annals of Applied Statistics.

[17]  D. Rubin,et al.  Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction , 2016 .

[18]  Costas Meghir,et al.  Subsidizing Vocational Training for Disadvantaged Youth in Colombia: Evidence from a Randomized Trial , 2011 .

[19]  S. Cole,et al.  Generalizing evidence from randomized clinical trials to target populations: The ACTG 320 trial. , 2010, American journal of epidemiology.

[20]  E. Oster,et al.  Unobservable Selection and Coefficient Stability: Theory and Validation , 2013 .

[21]  Patrick M. Kline,et al.  Evaluating Public Programs with Close Substitutes: The Case of Head Start , 2015 .

[22]  Jonathan Robinson,et al.  Why Don&Apos;T the Poor Save More? Evidence from Health Savings Experiments , 2011, The American economic review.

[23]  K. Muralidharan,et al.  Disrupting Education? Experimental Evidence on Technology-Aided Instruction in India , 2016, American Economic Review.

[24]  E. C. Fieller.  SOME PROBLEMS IN INTERVAL ESTIMATION , 1954 .

[25]  Eric T. Chyn.  Moved to Opportunity: The Long-Run Effect of Public Housing Demolition on Labor Market Outcomes of Children , 2016 .

[26]  E. Stuart,et al.  On the “Where” of Social Experiments: The Nature and Extent of the Generalizability Problem , 2016 .

[27]  Guido W. Imbens,et al.  Imposing Moment Restrictions from Auxiliary Data by Weighting , 1996, Review of Economics and Statistics.

[28]  Patrick M. Kline.  Oaxaca-Blinder as a Reweighting Estimator , 2011 .

[29]  Rachael Meager,et al.  Understanding the Average Impact of Microcredit Expansions: A Bayesian Hierarchical Analysis of Seven Randomized Experiments , 2019, American Economic Journal: Applied Economics.

[30]  Azeem M. Shaikh,et al.  Inference Under Covariate-Adaptive Randomization , 2015, Journal of the American Statistical Association.

[31]  Ana Santiago,et al.  Disrupting Education ? Experimental Evidence on Technology-Aided Instruction in India , 2019 .

[32]  Guido W. Imbens,et al.  External Validity in Fuzzy Regression Discontinuity Designs , 2014, Journal of Business & Economic Statistics.

[33]  Robert T. Jensen,et al.  Do Labor Market Opportunities Affect Young Women's Work and Family Decisions? Experimental Evidence from India , 2012 .

[34]  James J. Heckman,et al.  Econometric Evaluation of Social Programs, Part I: Causal Models, Structural Models and Econometric Policy Evaluation , 2007 .

[35]  Rajeev Dehejia,et al.  From Local to Global: External Validity in a Fertility Natural Experiment , 2015, Journal of Business & Economic Statistics.

[36]  Magne Mogstad,et al.  Beyond LATE with a discrete instrument. Heterogeneity in the quantity-quality interaction of children , 2012 .

[37]  Miikka A. T. Rokkanen,et al.  Exam Schools, Ability, and the Effects of Affirmative Action: Latent Factor Extrapolation in the Regression Discontinuity Design , 2015 .

[38]  D. Rubin,et al.  The central role of the propensity score in observational studies for causal effects , 1983 .

[39]  T. W. Anderson,et al.  Estimation of the Parameters of a Single Equation in a Complete System of Stochastic Equations , 1949 .

[40]  James J Heckman,et al.  Understanding Instrumental Variables in Models with Essential Heterogeneity , 2006, The Review of Economics and Statistics.

[41]  J. Hahn.  On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects , 1998 .

[42]  Sendhil Mullainathan,et al.  Site Selection Bias in Program Evaluation , 2012 .

[43]  Bo E. Honoré,et al.  The Empirical Content of the Roy Model , 1990 .

[44]  Christopher R. Taber,et al.  Selection on Observed and Unobserved Variables: Assessing the Effectiveness of Catholic Schools , 2000, Journal of Political Economy.

[45]  G. Imbens,et al.  Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score , 2000 .

[46]  Judd B. Kessler,et al.  The Effects of Youth Employment: Evidence from New York City Lotteries , 2016 .

[47]  Stephen H Bell,et al.  External Validity in Policy Evaluations that Choose Sites Purposively. , 2013, Journal of policy analysis and management : [the journal of the Association for Public Policy Analysis and Management].

[48]  Robert B. Olsen,et al.  Estimates of External Validity Bias When Impact Evaluations Select Sites Nonrandomly , 2016 .

[49]  Andres Santos,et al.  Using Instrumental Variables for Inference About Policy Relevant Treatment Effects , 2017 .

[50]  D. Horvitz,et al.  A Generalization of Sampling Without Replacement from a Finite Universe , 1952 .