How to See More in Observational Studies: Some New Quasi-Experimental Devices

In a well-conducted, slightly idealized, randomized experiment, the only explanation of an association between treatment and outcome is an effect caused by the treatment. However, this is not true in observational studies of treatment effects, in which treatment and outcomes may be associated because of some bias in the assignment of treatments to individuals. When added to the design of an observational study, quasi-experimental devices investigate empirically a particular rival explanation or counterclaim, often attempting to preempt anticipated counterclaims. This review has three parts: a discussion of the often misunderstood logic of quasi-experimental devices; a brief overview of the important work of Donald T. Campbell and his colleagues (excellent expositions of this work have been published elsewhere); and its main topic, descriptions and empirical examples of newer devices, including evidence factors, differential effects, and the computerized construction of quasi-experiments.

[1]  D. Campbell,et al.  Regression-Discontinuity Analysis: An Alternative to the Ex-Post Facto Experiment , 1960 .

[2]  Paul W. Holland,et al.  The sensitivity of linear regression coefficients' confidence limits to the omission of a confounder , 2009, 0905.3463.

[3]  J. S. Maritz,et al.  A note on exact robust confidence intervals for location , 1979 .

[4]  Chris J. Skinner,et al.  Variance estimation in the analysis of clustered longitudinal survey data , 2007 .

[5]  Chris J. Skinner,et al.  Estimating models for panel survey data under complex sampling , 2008 .

[6]  P. Rosenbaum,et al.  Contrasting Evidence Within and Between Institutions That Provide Treatment in an Observational Study of Alternate Forms of Anesthesia , 2012, Journal of the American Statistical Association.

[7]  P. Rosenbaum.  The Role of a Second Control Group in an Observational Study , 1987 .

[8]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[9]  K. Cummings,et al.  Reductions in tobacco smoke pollution and increases in support for smoke-free public places following the implementation of comprehensive smoke-free workplace legislation in the Republic of Ireland: findings from the ITC Ireland/UK Survey , 2006, Tobacco Control.

[10]  Daniel Levy,et al.  The Framingham Heart Study and the epidemiology of cardiovascular disease: a historical perspective , 2014, The Lancet.

[11]  Dylan S. Small,et al.  War and Wages : The Strength of Instrumental Variables and Their Sensitivity to Unobserved Biases , 2007 .

[12]  David R. Holtgrave,et al.  Alternatives to the randomized controlled trial. , 2008, American journal of public health.

[13]  B S Weir,et al.  Truncated product method for combining P‐values , 2002, Genetic epidemiology.

[14]  P. Diggle.  Analysis of Longitudinal Data , 1995 .

[15]  Bruce D. Meyer.  Natural and Quasi- Experiments in Economics , 1994 .

[16]  Paul R. Rosenbaum,et al.  Some Approximate Evidence Factors in Observational Studies , 2011 .

[17]  H. D. Patterson.  Sampling on Successive Occasions with Partial Replacement of Units , 1950 .

[18]  I. R. Savage,et al.  On the Independence of Tests of Randomness and Other Hypotheses , 1957 .

[19]  K E Warner,et al.  Smoking and lung cancer: an overview. , 1984, Cancer research.

[20]  P. Rosenbaum.  Discussing hidden bias in observational studies. , 1991, Annals of internal medicine.

[21]  Daniel S. Nagin,et al.  3. Bounding Disagreements about Treatment Effects: A Case Study of Sentencing and Recidivism , 1998 .

[22]  D. Cox.  Causality : some statistical aspects , 1992 .

[23]  Steven Hirschfeld,et al.  The National Children's Study--a proposed plan. , 2013, The New England journal of medicine.

[24]  R. Boruch,et al.  3 – Making the Case for Randomized Assignment to Treatments by Considering the Alternatives: Six Ways in Which Quasi-Experimental Evaluations In Compensatory Education Tend to Underestimate Effects , 1975 .

[25]  James O. Ramsay,et al.  Spatial spline regression models , 2013 .

[26]  Dylan S. Small,et al.  Stronger instruments via integer programming in an observational study of late preterm birth outcomes , 2013, 1304.4066.

[27]  Paul R. Rosenbaum,et al.  Matching for Balance, Pairing for Heterogeneity in an Observational Study of the Effectiveness of For-Profit and Not-For-Profit High Schools in Chile , 2014, 1404.3584.

[28]  Paul R. Rosenbaum,et al.  Robust, accurate confidence intervals with a weak instrument: quarter of birth and education , 2005 .

[29]  A. Velayati,et al.  Chromosomal aberrations and micronuclei in lymphocytes of patients before and after exposure to anti-tuberculosis drugs. , 2000, Mutagenesis.

[30]  Michael R Elliott,et al.  Bayesian inference for causal mediation effects using principal stratification with dichotomous mediators and outcomes. , 2010, Biostatistics.

[31]  Danny Pfeffermann,et al.  PARAMETRIC AND SEMI-PARAMETRIC ESTIMATION OF REGRESSION MODELS FITTED TO SURVEY DATA* , 2016 .

[32]  Adrian Waddell,et al.  Visual Clustering of High-dimensional Data by Navigating Low-dimensional Spaces , 2011 .

[33]  Lih-Yuan Deng,et al.  Orthogonal Arrays: Theory and Applications , 1999, Technometrics.

[34]  David R. Cox,et al.  Multiple randomizations , 2006 .

[35]  Katrina Armstrong,et al.  An Algorithm for Optimal Tapered Matching, With Application to Disparities in Survival , 2008, Journal of computational and graphical statistics : a joint publication of American Statistical Association, Institute of Mathematical Statistics, Interface Foundation of North America.

[36]  D. Wolfe.  Some General Results about Uncorrelated Statistics , 1973 .

[37]  F. R. Rosendaal,et al.  Experience with multiple control groups in a large population-based case–control study on genetic and environmental risk factors , 2010, European Journal of Epidemiology.

[38]  Paul R. Rosenbaum,et al.  Replicating Effects and Biases , 2001 .

[39]  Bin Nan,et al.  A Hot‐Deck Multiple Imputation Procedure for Gaps in Longitudinal Recurrent Event Histories , 2011, Biometrics.

[40]  Chris J. Skinner,et al.  Random effects models for longitudinal survey data , 2003 .

[41]  Paul R Rosenbaum,et al.  Optimal Pair Matching With Two Control Groups , 2004 .

[42]  P. Rosenbaum.  Sensitivity analysis for matching with multiple controls , 1988 .

[43]  J. Marden,et al.  Use of Nested Orthogonal Contrasts in Analyzing Rank Data , 1992 .

[44]  Danyu Lin,et al.  On fitting Cox's proportional hazards models to survey data , 2000 .

[45]  P. Rosenbaum.  Design of Observational Studies , 2009, Springer Series in Statistics.

[46]  J. Krosnick.  Response strategies for coping with the cognitive demands of attitude measures in surveys , 1991 .

[47]  James F. Thrasher,et al.  Stability of cigarette consumption over time among continuing smokers: a latent growth curve analysis. , 2012, Nicotine & tobacco research : official journal of the Society for Research on Nicotine and Tobacco.

[48]  Dylan S. Small,et al.  Using Split Samples and Evidence Factors in an Observational Study of Neonatal Outcomes , 2011 .

[49]  P. Rosenbaum.  Testing hypotheses in order , 2008 .

[50]  Jeffrey H Silber,et al.  Time to send the preemie home? Additional maturity at discharge and subsequent health care costs and outcomes. , 2009, Health services research.

[51]  Jean-Paul Fox,et al.  Longitudinal measurement in health‐related surveys. A Bayesian joint growth model for multivariate ordinal responses , 2013, Statistics in medicine.

[52]  David A. Jaeger,et al.  Problems with Instrumental Variables Estimation when the Correlation between the Instruments and the Endogenous Explanatory Variable is Weak , 1995 .

[53]  D. Campbell.  Factors relevant to the validity of experiments in social settings. , 1957, Psychological bulletin.

[54]  S. Galea,et al.  Trajectories of posttraumatic stress symptomatology in older persons affected by a large-magnitude disaster. , 2013, Journal of psychiatric research.

[55]  P. Rosenbaum,et al.  Amplification of Sensitivity Analysis in Matched Observational Studies , 2009, Journal of the American Statistical Association.

[56]  M. Norton,et al.  Reduced prevalence of AD in users of NSAIDs and H2 receptor antagonists , 2000, Neurology.

[57]  Using Longitudinal Surveys to Evaluate Interventions , 2009 .

[58]  P. Holland.  CAUSAL INFERENCE, PATH ANALYSIS AND RECURSIVE STRUCTURAL EQUATIONS MODELS , 1988 .

[59]  G. W. Imbens.  Sensitivity to Exogeneity Assumptions in Program Evaluation , 2003 .

[60]  D. Campbell,et al.  EXPERIMENTAL AND QUASI-EXPERIMENT Al DESIGNS FOR RESEARCH , 2012 .

[61]  J. Hahn,et al.  IDENTIFICATION AND ESTIMATION OF TREATMENT EFFECTS WITH A REGRESSION-DISCONTINUITY DESIGN , 2001 .

[62]  S. Cannistra.  Gynecologic oncology or medical oncology: what's in a name? , 2007, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[63]  J. Rao,et al.  Using Marginal Mean Models for Data from Longitudinal Surveys with a Complex Design: Some Advances in Methods , 2009 .

[64]  M. Sobel,et al.  Identification Problems in the Social Sciences. , 1996 .

[65]  P. Rosenbaum.  Using Differential Comparisons in Observational Studies , 2013 .

[66]  R. A. Fisher,et al.  Design of Experiments , 1936 .

[67]  M. Zelen.  A new design for randomized clinical trials. , 1979, The New England journal of medicine.

[68]  Robert D Gibbons,et al.  Post-approval drug safety surveillance. , 2010, Annual review of public health.

[69]  Jeffrey H Silber,et al.  Characteristics associated with differences in survival among black and white women with breast cancer. , 2013, JAMA.

[70]  E. C. Hammond,et al.  SMOKING IN RELATION TO MORTALITY AND MORBIDITY. FINDINGS IN FIRST THIRTY-FOUR MONTHS OF FOLLOW-UP IN A PROSPECTIVE STUDY STARTED IN 1959. , 1964, Journal of the National Cancer Institute.

[71]  B Rosner,et al.  Postmenopausal estrogen and progestin use and the risk of cardiovascular disease. , 1996, The New England journal of medicine.

[72]  J. Lawless,et al.  Estimation of finite population duration distributions from longitudinal survey panels with intermittent followup , 2013, Lifetime Data Analysis.

[73]  Dylan S. Small,et al.  Building a Stronger Instrument in an Observational Study of Perinatal Care for Premature Infants , 2010 .

[74]  R. Peto.  THE HORSE-RACING EFFECT , 1981, The Lancet.

[75]  Some Nonparametric Tests of Randomness , 1974 .

[76]  Andrew Gelman,et al.  Struggles with survey weighting and regression modeling , 2007, 0710.5005.

[77]  R. PaulR..  Stability in the Absence of Treatment , 2004 .

[78]  Barbara A. Bailar,et al.  The Effects of Rotation Group Bias on Estimates from Panel Surveys , 1975 .

[79]  James M. Robins,et al.  Observational Studies Analyzed Like Randomized Experiments: An Application to Postmenopausal Hormone Therapy and Coronary Heart Disease , 2008, Epidemiology.

[80]  S. Resnick.  A Probability Path , 1999 .

[81]  M. Zanna,et al.  Establishing a causal chain: why experiments are often more effective than mediational analyses in examining psychological processes. , 2005, Journal of personality and social psychology.

[82]  Cecilia Elena Rouse,et al.  Democratization or Diversion? The Effect of Community Colleges on Educational Attainment , 1995 .

[83]  Joshua D. Angrist,et al.  Identification of Causal Effects Using Instrumental Variables , 1993 .

[84]  P. Holland.  Causal Inference, Path Analysis and Recursive Structural Equations Models. Program Statistics Research, Technical Report No. 88-81. , 1988 .

[85]  Govind S. Mudholkar,et al.  A class of tests for equality of ordered means , 1989 .

[86]  W. G. Cochran.  The Planning of Observational Studies of Human Populations , 1965 .

[87]  J. Robins,et al.  Sensitivity Analysis for Selection bias and unmeasured Confounding in missing Data and Causal inference models , 2000 .

[88]  Danny Pfeffermann,et al.  Multilevel modelling of complex survey longitudinal data with time varying random effects , 2000 .

[89]  Eiliv Lund,et al.  Reduced breast cancer mortality among fishermen's wives in Norway , 1993, Cancer Causes & Control.

[90]  T. J. Terpstra,et al.  The asymptotic normality and consistency of kendall's test against trend, when ties are present in one ranking , 1952 .

[91]  E. C. Hammond,et al.  Smoking in Relation to Mortality and Morbidity. , 1965 .

[92]  D. Campbell,et al.  Evolving Methods for Enhancing Validity@@@Methodology and Epistemology for Social Science: Selected Papers , 1990 .

[93]  Dylan S. Small,et al.  Effect Modification and Design Sensitivity in Observational Studies , 2013 .

[94]  J. Thigpen.  Does Ovarian Cancer Treatment and Survival Differ by the Specialty Providing Chemotherapy , 2008 .

[95]  Elizabeth A Stuart,et al.  Should epidemiologists be more sensitive to design sensitivity? , 2013, Epidemiology.

[96]  Daniel S. Nagin,et al.  Evidence and Public Policy , 2013 .

[97]  Robert V. Hogg,et al.  Certain Uncorrelated and Independent Rank Statistics , 1971 .

[98]  Tyler J. VanderWeele,et al.  Conceptual issues concerning mediation, interventions and composition , 2009 .

[99]  T. DiPrete,et al.  7. Assessing Bias in the Estimation of Causal Effects: Rosenbaum Bounds on Matching Estimators and Instrumental Variables Estimation with Imperfect Instruments , 2004 .

[100]  Donald B. Rubin,et al.  BEST PRACTICES IN QUASI- EXPERIMENTAL DESIGNS Matching Methods for Causal Inference , 2007 .

[101]  Daniel Manrique-Vallier,et al.  Longitudinal Mixed Membership Trajectory Models for Disability Survey Data. , 2013, The annals of applied statistics.

[102]  R. Little.  Pattern-Mixture Models for Multivariate Incomplete Data , 1993 .

[103]  P. Rosenbaum.  Impact of Multiple Matched Controls on Design Sensitivity in Observational Studies , 2013, Biometrics.

[104]  P. Rosenbaum,et al.  Using the Exterior Match to Compare Two Entwined Matched Control Groups , 2013 .

[105]  Rocío Titiunik,et al.  Enhancing a geographic regression discontinuity design through matching to estimate the effect of ballot initiatives on voter turnout , 2015 .

[106]  P. Rosenbaum.  An exact adaptive test with superior design sensitivity in an observational study of treatments for ovarian cancer , 2012, 1203.3672.

[107]  Xinyi Xu,et al.  Optimal Nonbipartite Matching and Its Statistical Applications , 2011, The American statistician.

[108]  Alan E Hubbard,et al.  Simulation methods to estimate design power: an overview for applied research , 2011, BMC medical research methodology.

[109]  E. C. Hammond,et al.  Smoking and lung cancer: recent evidence and a discussion of some questions. , 1959, Journal of the National Cancer Institute.

[110]  S. Zeger,et al.  Longitudinal data analysis using generalized linear models , 1986 .

[111]  P. Rosenbaum.  Differential effects and generic biases in observational studies , 2006 .

[112]  Donald B. Rubin,et al.  Matching With Multiple Control Groups With Adjustment for Group Differences , 2008 .

[113]  James F. Thrasher,et al.  Mediational pathways of the impact of cigarette warning labels on quit attempts. , 2014, Health psychology : official journal of the Division of Health Psychology, American Psychological Association.

[114]  D. Rubin,et al.  Assessing Sensitivity to an Unobserved Binary Covariate in an Observational Study with Binary Outcome , 1983 .

[115]  Changbao Wu,et al.  The pseudo‐GEE approach to the analysis of longitudinal surveys , 2010 .

[116]  Elizabeth A. Stuart,et al.  An Introduction to Sensitivity Analysis for Unobserved Confounding in Nonexperimental Prevention Research , 2013, Prevention Science.

[117]  G. Catlin,et al.  Sample design of the national population health survey. , 1995, Health reports.

[118]  Analysing ordinal longitudinal survey data: Generalised estimating equations approach , 2000 .

[119]  Charles Kooperberg,et al.  Risks and benefits of estrogen plus progestin in healthy postmenopausal women: principal results From the Women's Health Initiative randomized controlled trial. , 2002, JAMA.

[120]  P. Rosenbaum.  Evidence factors in observational studies , 2010 .

[121]  D. Rubin.  Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[122]  Dylan S. Small,et al.  Using the Cross-Match Test to Appraise Covariate Balance in Matched Pairs , 2010 .

[123]  Paul R. Rosenbaum,et al.  Testing one hypothesis twice in observational studies , 2012 .

[124]  A. Karr,et al.  Combining cohorts in longitudinal surveys , 2013 .

[125]  Paul R Rosenbaum,et al.  Sensitivity Analysis for m‐Estimates, Tests, and Confidence Intervals in Matched Observational Studies , 2007, Biometrics.

[126]  Paul Smith,et al.  Sample design for longitudinal surveys , 2009 .

[127]  Susana Rubin-Bleuer.  The proportional hazards model for survey data from independent and clustered super-populations , 2011, J. Multivar. Anal..

[128]  A. Rotnitzky,et al.  Missing Data in Longitudinal Studies: Strategies for Bayesian Modeling and Sensitivity Analysis by DANIELS, M. J. and HOGAN, J. W , 2009 .

[129]  Dylan S. Small,et al.  War and Wages , 2008 .