Unbiased Causal Inference From an Observational Study: Results of a Within-Study Comparison

Adjustment methods such as propensity scores and analysis of covariance are often used for estimating treatment effects in nonexperimental data. Shadish, Clark, and Steiner used a within-study comparison to test how well these adjustments work in practice. They randomly assigned participating students to a randomized or nonrandomized experiment. Treatment effects were then estimated in the experiment and compared to the adjusted nonexperimental estimates. Most of the selection bias in the nonexperiment was reduced. The present study replicates the study of Shadish et al. despite some differences in design and in the size and direction of the initial bias. The results show that the selection of covariates matters considerably for bias reduction in nonexperiments but that the choice of analysis matters less.

[1]  David E. Booth,et al.  Analysis of Incomplete Multivariate Data , 2000, Technometrics.

[2]  D. Rubin [On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9.] Comment: Neyman (1923) and Causal Inference in Experiments and Observational Studies , 1990 .

[3]  D. Watson,et al.  Positive and negative affectivity and their relation to anxiety and depressive disorders. , 1988, Journal of abnormal psychology.

[4]  D. Rubin ASSIGNMENT TO TREATMENT GROUP ON THE BASIS OF A COVARIATE , 1976 .

[5]  Carl-Walter Kohlmann,et al.  Untersuchungen mit einer deutschen Version der "Positive and Negative Affect Schedule" (PANAS). , 1996 .

[6]  Donald B Rubin,et al.  On principles for modeling propensity scores in medical research , 2004, Pharmacoepidemiology and drug safety.

[7]  D. Rubin Using Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation , 2001, Health Services and Outcomes Research Methodology.

[8]  Peter M. Steiner,et al.  The importance of covariate selection in controlling for selection bias in observational studies. , 2010, Psychological methods.

[9]  J. Schafer,et al.  Average causal effects from nonrandomized studies: a practical guide and simulated example. , 2008, Psychological methods.

[10]  J. Lunceford,et al.  Stratification and weighting via the propensity score in estimation of causal treatment effects: a comparative study , 2004, Statistics in medicine.

[11]  W. G. Cochran,et al.  Controlling Bias in Observational Studies: A Review. , 1974 .

[12]  Donald B. Rubin,et al.  Comment: The Design and Analysis of Gold Standard Randomized Experiments , 2008 .

[13]  T. Speed,et al.  On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9 , 1990 .

[14]  D. Rubin Matched Sampling for Causal Effects: The Use of Matched Sampling and Regression Adjustment to Remove Bias in Observational Studies , 1973 .

[15]  Walter Zucchini,et al.  Model Selection , 2011, International Encyclopedia of Statistical Science.

[16]  Joshua D. Angrist,et al.  Identification of Causal Effects Using Instrumental Variables , 1993 .

[17]  P. Holland Statistics and Causal Inference , 1985 .

[18]  G. Imbens,et al.  Estimation of Causal Effects using Propensity Score Weighting: An Application to Data on Right Heart Catheterization , 2001, Health Services and Outcomes Research Methodology.

[19]  Juan-José Díaz,et al.  An Assessment of Propensity Score Matching as a Nonexperimental Impact Estimator , 2005, The Journal of Human Resources.

[20]  D. Watson,et al.  Development and validation of brief measures of positive and negative affect: the PANAS scales. , 1988, Journal of personality and social psychology.

[21]  Donald B. Rubin,et al.  Bayesian Inference for Causal Effects: The Role of Randomization , 1978 .

[22]  D. Rubin Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[23]  Gary King,et al.  Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference , 2007, Political Analysis.

[24]  D. Rubin Matched Sampling for Causal Effects: Matching to Remove Bias in Observational Studies , 1973 .

[25]  R. Lalonde Evaluating the Econometric Evaluations of Training Programs with Experimental Data , 1984 .

[26]  Arthur S. Goldberger,et al.  Selection bias in evaluating treatment effects: Some formal illustrations , 2008 .

[27]  G. Imbens,et al.  On the Failure of the Bootstrap for Matching Estimators , 2006 .

[28]  Vivian C. Wong,et al.  Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within‐study comparisons , 2008 .

[29]  Peter M. Steiner,et al.  Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments , 2008 .

[30]  D. Rubin,et al.  Reducing Bias in Observational Studies Using Subclassification on the Propensity Score , 1984 .

[31]  D B Rubin,et al.  Matching using estimated propensity scores: relating theory to practice. , 1996, Biometrics.

[32]  H. Harman,et al.  Kit of factor-referenced cognitive tests , 1976 .

[33]  T. Shakespeare,et al.  Observational Studies , 2003 .

[34]  D. Rubin,et al.  The central role of the propensity score in observational studies for causal effects , 1983 .

[35]  J. Robins,et al.  Semiparametric Efficiency in Multivariate Regression Models with Missing Data , 1995 .

[36]  Joseph Kang,et al.  Demystifying Double Robustness: A Comparison of Alternative Strategies for Estimating a Population Mean from Incomplete Data , 2007, 0804.2958.

[37]  William M. K. Trochim,et al.  Research Design for Program Evaluation: The Regression-Discontinuity Approach , 1984 .

[38]  Gary King,et al.  Misunderstandings between experimentalists and observationalists about causal inference , 2008 .

[39]  W. G. Cochran The effectiveness of adjustment by subclassification in removing bias in observational studies. , 1968, Biometrics.

[40]  D. Rubin For objective causal inference, design trumps analysis , 2008, 0811.1640.

[41]  Steven Glazerman,et al.  Nonexperimental Versus Experimental Estimates of Earnings Impacts , 2003 .

[42]  D B Rubin,et al.  Estimating the causal effects of smoking , 2001, Statistics in medicine.