From SATE to PATT : Combining Experimental with Observational Studies to Estimate Population Treatment Effects ∗

Randomised controlled trials (RCTs) can provide unbiased estimates of sample average treatment effects. However, a common concern is that RCTs often fail to provide unbiased estimates of population average treatment effects. We derive the assumptions for identifying population average treatment effects from RCTs. We provide a set of placebo tests, which formally follow from the identifying assumptions, that can assess whether the assumptions hold. We offer new research designs for estimating population effects that use non-random studies (NRSs) to adjust the RCT data. One design does not require a selection on observables assumption. We apply our approach to a cost-effectiveness analysis of a controversial clinical intervention, Pulmonary Artery Catheterization (PAC).

[1]  E. Jaynes.  Information Theory and Statistical Mechanics , 1957 .

[2]  S. Kullback,et al.  Contingency tables with given marginals. , 1968, Biometrika.

[3]  L. Hansen.  Large Sample Properties of Generalized Method of Moments Estimators , 1982 .

[4]  Joseph P. Romano,et al.  Large Sample Confidence Regions Based on Subsamples under Minimal Assumptions , 1994 .

[5]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[6]  William A. Knaus,et al.  The effectiveness of right heart catheterization in the initial care of critically ill patients. SUPPORT Investigators. , 1996, Journal of the American Medical Association (JAMA).

[7]  S. Yusuf,et al.  Overcoming the limitations of current meta-analysis of randomised controlled trials , 1998, The Lancet.

[8]  James J. Heckman,et al.  Characterizing Selection Bias Using Experimental Data , 1998 .

[9]  B O'Brien,et al.  Statistical analysis of cost effectiveness data. , 1999, The Journal of rheumatology.

[10]  D. Carnall.  Randomised controlled trials , 1999, BMJ.

[11]  Guido W. Imbens,et al.  Imposing Moment Restrictions from Auxiliary Data by Weighting , 1996, Review of Economics and Statistics.

[12]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[13]  J. Dalen.  The pulmonary artery catheter-friend, foe, or accomplice? , 2001, JAMA.

[14]  Catherine P. Bradshaw,et al.  The use of propensity scores to assess the generalizability of results from randomized trials , 2011, Journal of the Royal Statistical Society. Series A,.

[15]  D Y Lin,et al.  Incremental net benefit in randomized clinical trials , 2001, Statistics in medicine.

[16]  Andrew R Willan,et al.  Incremental net benefit in randomized clinical trials with quality‐adjusted survival , 2003, Statistics in medicine.

[17]  J. Zivin,et al.  Effectiveness and cost-effectiveness of four treatment modalities for substance disorders: a propensity score analysis. , 2003, Health services research.

[18]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[19]  James A. Russell,et al.  FEATURE ARTICLES , 2004, Critical care medicine.

[20]  Andrew R Willan,et al.  Regression methods for covariate adjustment and subgroup analysis for non-censored cost-effectiveness data. , 2004, Health economics.

[21]  Kathy Rowan,et al.  Case mix, outcome and length of stay for admissions to adult, general critical care units in England, Wales and Northern Ireland: the Intensive Care National Audit & Research Centre Case Mix Programme Database , 2004, Critical care.

[22]  Kathy Rowan,et al.  The incremental cost effectiveness of withdrawing pulmonary artery catheters from routine use in critical care , 2005, Applied health economics and health policy.

[23]  P. Rothwell,et al.  External validity of randomised controlled trials: “To whom do the results of this trial apply?” , 2005, The Lancet.

[24]  Simon G Thompson,et al.  Methods for incorporating covariate adjustment, subgroup analysis and between-centre differences into cost-effectiveness evaluations. , 2005, Health economics.

[25]  Sander Greenland,et al.  Multiple‐bias modelling for analysis of observational data , 2005 .

[26]  N. Mitra,et al.  A propensity score approach to estimating the cost-effectiveness of medical therapies from observational data. , 2005, Health economics.

[27]  V. J. Hotz,et al.  Predicting the efficacy of future training programs using past experiences at other locations , 2005 .

[28]  Didier Payen,et al.  Use of the pulmonary artery catheter is not associated with worse outcome in the ICU. , 2005, Chest.

[29]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[30]  G. Imbens,et al.  On the Failure of the Bootstrap for Matching Estimators , 2006 .

[31]  James J Heckman,et al.  Understanding Instrumental Variables in Models with Essential Heterogeneity , 2006, The Review of Economics and Statistics.

[32]  E. Stuart,et al.  Misunderstandings among Experimentalists and Observationalists about Causal Inference , 2007 .

[33]  Andy Liaw,et al.  Classification and Regression by randomForest , 2007 .

[34]  P. Bickel,et al.  ON THE CHOICE OF m IN THE m OUT OF n BOOTSTRAP AND CONFIDENCE BOUNDS FOR EXTREMA , 2008 .

[35]  M. Singer,et al.  Post hoc insights from PAC-Man—The U.K. pulmonary artery catheter trial* , 2008, Critical care medicine.

[36]  Jasjeet S. Sekhon,et al.  A New Non-Parametric Matching Method for Bias Adjustment with Applications to Economic Evaluations , 2008 .

[37]  William Gardner,et al.  Generalizing from clinical trial data: A case study. The risk of suicidality among pediatric antidepressant users , 2008, Statistics in medicine.

[38]  David J Spiegelhalter,et al.  Bias modelling in evidence synthesis , 2009, Journal of the Royal Statistical Society. Series A,.

[39]  A. Deaton.  Instruments of Development: Randomization in the Tropics, and the Search for the Elusive Keys to Economic Development , 2009 .

[40]  Jasjeet S. Sekhon,et al.  Opiates for the Matches: Matching Methods for Causal Inference , 2009 .

[41]  Stephen R Cole,et al.  The consistency statement in causal inference: a definition or an assumption? , 2009, Epidemiology.

[42]  L. Hedges,et al.  The Handbook of Research Synthesis and Meta-Analysis , 2009 .

[43]  G. Imbens,et al.  Better Late than Nothing: Some Comments on Deaton (2009) and Heckman and Urzua (2009) , 2009 .

[44]  S. Wellek.  Testing Statistical Hypotheses of Equivalence and Noninferiority , 2010 .

[45]  James J Heckman,et al.  Comparing IV with Structural Models: What Simple IV Can and Cannot Identify , 2009, Journal of econometrics.

[46]  S. Cole,et al.  Generalizing evidence from randomized clinical trials to target populations: The ACTG 320 trial. , 2010, American journal of epidemiology.

[47]  D. Green,et al.  Modeling heterogeneous treatment effects in large-scale experiments using Bayesian Additive Regression Trees , 2010 .

[48]  H. Chipman,et al.  BART: Bayesian Additive Regression Trees , 2008, 0806.3286.

[49]  E. Tamer,et al.  Using Observational vs. Randomized Controlled Trial Data to Learn About Treatment Effects , 2011 .

[50]  Kristin E. Porter,et al.  The Relative Performance of Targeted Maximum Likelihood Estimators , 2011, The international journal of biostatistics.

[51]  H. Allcott,et al.  Social Norms and Energy Conservation , 2011 .

[52]  Evgueni A. Haroutunian,et al.  Information Theory and Statistics , 2011, International Encyclopedia of Statistical Science.

[53]  S. Mullainathan,et al.  External Validity and Partner Selection Bias , 2012 .

[54]  J. Sekhon,et al.  Genetic Matching for Estimating Causal Effects: A General Multivariate Matching Method for Achieving Balance in Observational Studies , 2006, Review of Economics and Statistics.

[55]  Luke W. Miratrix,et al.  Adjusting treatment effect estimates by post‐stratification in randomized experiments , 2013 .

[56]  T. Roberts,et al.  Centre Selection for Clinical Trials and the Generalisability of Results: A Mixed Methods Study , 2013, PloS one.

[57]  Guido Imbens,et al.  Site Selection Bias in Program Evaluation , 2014 .

[58]  David Card,et al.  WORKING PAPER SERIES BETTER LATE THAN NOTHING : SOME COMMENTS ON DEATON ( 2009 ) AND HECKMAN AND URZUA , 2022 .