How to Examine External Validity within an Experiment

A fundamental concern for researchers who analyze and design experiments is that the experimental result might not be externally valid for all policies. Researchers often attempt to assess external validity by comparing data from an experiment to external data. In this essay, I discuss approaches from the treatment effects literature that researchers can use to begin the examination of external validity internally, within the data from a single experiment. I focus on presenting the approaches simply using figures. Institutional subscribers to the NBER working paper series, and residents of developing countries may download this paper without additional charge at www.nber.org.

[1]  J. Angrist.  Treatment Effect Heterogeneity in Theory and Practice , 2003 .

[2]  J. Heckman,et al.  Policy-Relevant Treatment Effects , 2001 .

[3]  Does Disability Insurance Receipt Discourage Work? Using Examiner Assignment to Estimate Causal Effects of SSDI Receipt , 2012 .

[4]  James J. Heckman,et al.  Econometric Evaluation of Social Programs, Part II: Using the Marginal Treatment Effect to Organize Alternative Econometric Estimators to Evaluate Social Programs, and to Forecast their Effects in New Environments , 2007 .

[5]  A. Roy.  Some thoughts on the distribution of earnings , 1951 .

[6]  Amanda E. Kowalski.  Behavior within a Clinical Trial and Implications for Mammography Guidelines , 2018, The Review of economic studies.

[7]  Amanda E. Kowalski.  Doing More When You&Apos;Re Running Late: Applying Marginal Treatment Effect Methods to Examine Treatment Effect Heterogeneity in Experiments , 2016 .

[8]  A simple test for the ignorability of non-compliance in experiments , 2013 .

[9]  J J Heckman,et al.  Local instrumental variables and latent variable models for identifying and bounding treatment effects. , 1999, Proceedings of the National Academy of Sciences of the United States of America.

[10]  Amanda E. Kowalski.  Reconciling Seemingly Contradictory Results from the Oregon Health Insurance Experiment and the Massachusetts Health Reform , 2018, Review of Economics and Statistics.

[11]  Robert A. Moffitt,et al.  The Estimation of Wage Gains and Welfare Gains in Self-selection , 1987 .

[12]  Dan A. Black,et al.  Simple Tests for Selection Bias: Learning More from Instrumental Variables , 2015, SSRN Electronic Journal.

[13]  D. Rubin,et al.  Estimating Outcome Distributions for Compliers in Instrumental Variables Models , 1997 .

[14]  J. Angrist,et al.  Identification and Estimation of Local Average Treatment Effects , 1995 .

[15]  Patrick M. Kline,et al.  Evaluating Public Programs with Close Substitutes: The Case of Head Start , 2015 .

[16]  Anders Björklund,et al.  Estimation of Wage Gains and Welfare Gains from Self-Selection Models , 1983 .

[17]  Alberto Abadie.  Semiparametric instrumental variable estimation of treatment response models , 2003 .

[18]  Erik Snowberg,et al.  Selective Trials: A Principal-Agent Approach to Randomized Controlled Experiments , 2012 .

[19]  W. Greene.  Sample Selection Bias as a Specification Error: Comment , 1981 .

[20]  C. Manski.  Nonparametric Bounds on Treatment Effects , 1989 .

[21]  J. Heckman.  Sample selection bias as a specification error , 1979 .

[22]  Patrick M. Kline,et al.  On Heckits, Late, and Numerical Equivalence , 2017, Econometrica.

[23]  J. Angrist,et al.  Extrapolate-Ing: External Validity and Overidentification in the Late Framework , 2010 .

[24]  G. Fischer,et al.  Eliciting and Utilizing Willingness to Pay: Evidence from Field Trials in Northern Ghana , 2015, Journal of Political Economy.

[25]  Amanda E. Kowalski.  Extrapolation using Selection and Moral Hazard Heterogeneity from within the Oregon Health Insurance Experiment , 2018 .

[26]  Edward Vytlacil,et al.  Local Instrumental Variables , 2000 .

[27]  T. Cornelissen,et al.  Who Benefits from Universal Child Care? Estimating Marginal Returns to Early Child Care Attendance , 2018, Journal of Political Economy.

[28]  A. Basu.  Welfare Implications of Learning Through Solicitation Versus Diversification in Health Care , 2014, Journal of health economics.

[29]  V. J. Hotz,et al.  Predicting the efficacy of future training programs using past experiences at other locations , 2005 .

[30]  L. Einav,et al.  Estimating Welfare in Insurance Markets Using Variation in Prices , 2008, The quarterly journal of economics.

[31]  Magne Mogstad,et al.  Beyond LATE with a discrete instrument. Heterogeneity in the quantity-quality interaction of children , 2012 .

[32]  Chen Meng.  The Effect of Age at School Entry on Educational Attainment and Labor Market Outcomes: Evidence from China , 2017 .

[33]  R. Olsen,et al.  A Least Squares Correction for Selectivity Bias , 1980 .

[34]  J. Heckman,et al.  Making the Most out of Programme Evaluations and Social Experiments: Accounting for Heterogeneity in Programme Impacts , 1997 .

[35]  E. Vytlacil.  Independence, Monotonicity, and Latent Index Models: An Equivalence Result , 2002 .

[36]  J. Angrist,et al.  The Effect of Age at School Entry on Educational Attainment: An Application of Instrumental Variables with Moments from Two Samples , 1990 .

[37]  Guido W. Imbens,et al.  External Validity in Fuzzy Regression Discontinuity Designs , 2014, Journal of Business & Economic Statistics.

[38]  Edward Vytlacil,et al.  Estimating Marginal Returns to Education , 2010, The American economic review.

[39]  Dylan S Small,et al.  Using an instrumental variable to test for unmeasured confounding , 2014, Statistics in medicine.

[40]  Joshua D. Angrist,et al.  Lifetime Earnings and the Vietnam Era Draft Lottery: Evidence from Social Security Administrative Records , 1990 .

[41]  Andrés Santos,et al.  Using Instrumental Variables for Inference About Policy Relevant Treatment Effects , 2017 .

[42]  A. Wald.  The Fitting of Straight Lines if Both Variables are Subject to Error , 1940 .

[43]  T. Mroz,et al.  The Sensitivity of an Empirical Model of Married Women's Hours of Work to Economic and Statistical Assumptions , 1987 .

[44]  Magne Mogstad,et al.  Beyond LATE with a Discrete Instrument , 2017, Journal of Political Economy.

[45]  Jesse M. Shapiro,et al.  Can Higher Prices Stimulate Product Use? Evidence from a Field Experiment in Zambia , 2010 .

[46]  S. Lee,et al.  Estimating distributions of potential outcomes using local instrumental variables with an application to changes in college enrollment and wage inequality , 2009 .

[47]  Joshua D. Angrist,et al.  Identification of Causal Effects Using Instrumental Variables , 1993 .

[48]  J. Hausman.  Specification tests in econometrics , 1978 .

[49]  James J. Heckman,et al.  Substitution and Dropout Bias in Social Experiments: A Study of an Influential Social Experiment , 2000 .

[50]  Alberto Abadie.  Bootstrap Tests for Distributional Treatment Effects in Instrumental Variable Models , 2002 .

[51]  Andres Santos,et al.  Using Instrumental Variables for Inference About Policy Relevant Treatment Effects , 2017 .

[52]  J. Heckman.  The Common Structure of Statistical Models of Truncation, Sample Selection and Limited Dependent Variables and a Simple Estimator for Such Models , 1976 .