Effect estimates in randomized trials and observational studies: comparing apples with apples.

Effect estimates from randomized trials and observational studies may not be directly comparable because of differences in study design, other than randomization, and in data analysis. We propose a three-step procedure to facilitate meaningful comparisons of effect estimates from randomized trials and observational studies: 1) harmonization of the study protocol (eligibility criteria, treatment strategies, outcome, start and end of follow-up, causal contrast) so that the studies target the same causal effect, 2) harmonization of the data analysis to estimate the causal effect, and 3) sensitivity analyses to investigate the impact of discrepancies that could not be accounted for in the harmonization process. To illustrate our approach, we compared estimates of the effect of immediate with deferred initiation of antiretroviral therapy in individuals positive to the human immunodeficiency virus from the START randomized trial and the observational HIV-CAUSAL Collaboration.

[1]  M. Mahy,et al.  Opportunistic infections and AIDS malignancies early after initiating combination antiretroviral therapy in high-income countries , 2014, AIDS.

[2]  M. Battegay,et al.  Virologic and Immunologic Responses in Treatment-Naive Patients to Ritonavir-Boosted Atazanavir or Efavirenz With a Common Backbone , 2014, HIV clinical trials.

[3]  Pam Sonnenberg,et al.  How soon after infection with HIV does the risk of tuberculosis start to increase? A retrospective cohort study in South African gold miners. , 2005, The Journal of infectious diseases.

[4]  Haitao Chu,et al.  The per-protocol effect of immediate versus deferred antiretroviral therapy initiation , 2016, AIDS.

[5]  G. Guyatt,et al.  Stopping randomized trials early for benefit and estimation of treatment effects: systematic review and meta-regression analysis. , 2010, JAMA.

[6]  Lei Wang,et al.  Prevention of HIV-1 infection with early antiretroviral therapy. , 2011, The New England journal of medicine.

[7]  S. Buchbinder,et al.  Early versus deferred antiretroviral therapy for HIV. , 2009, The New England journal of medicine.

[8]  H. Bang,et al.  Early versus standard antiretroviral therapy for HIV-infected adults in Haiti. , 2010, The New England journal of medicine.

[9]  M. T. Paixão,et al.  Expanded European AIDS case definition , 1993, The Lancet.

[10]  Jessica G. Young,et al.  Comparative effectiveness of immediate antiretroviral therapy versus CD4-based initiation in HIV-positive individuals in high-income countries: observational cohort study. , 2015, The lancet. HIV.

[11]  J. Concato,et al.  Randomized, controlled trials, observational studies, and the hierarchy of research designs. , 2000, The New England journal of medicine.

[12]  James M Robins,et al.  Using Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available. , 2016, American journal of epidemiology.

[13]  M. Hernán,et al.  Beyond the intention-to-treat in comparative effectiveness research , 2012, Clinical trials.

[14]  H. Lane,et al.  Initiation of Antiretroviral Therapy in Early Asymptomatic HIV Infection. , 2015, The New England journal of medicine.

[15]  G. Rutherford,et al.  Early initiation of antiretroviral therapy in HIV-infected adults and adolescents: a systematic review , 2014, AIDS.

[16]  C. Morrison,et al.  Risk of tuberculosis following HIV seroconversion in high-income countries , 2012, Thorax.

[17]  Michael J Silverberg,et al.  Effect of early versus deferred antiretroviral therapy for HIV on survival. , 2009, The New England journal of medicine.

[18]  J. Ioannidis,et al.  Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials: meta-epidemiological survey , 2016, British Medical Journal.

[19]  S. Cole,et al.  Generalizing Randomized Clinical Trial Results: Implementation and Challenges Related to Missing Data in the Target Population , 2018, American journal of epidemiology.

[20]  M. Hernán,et al.  Antiretroviral penetration into the CNS and incidence of AIDS-defining neurologic conditions , 2014, Neurology.

[21]  Stephen R Cole,et al.  Timing of initiation of antiretroviral therapy in AIDS-free HIV-1-infected patients: a collaborative analysis of 18 HIV cohort studies , 2009, The Lancet.

[22]  J. Pearl,et al.  Causal Inference , 2011, Twenty-one Mental Models That Can Change Policing.

[23]  S. Cole,et al.  Timing of HAART initiation and clinical outcomes in human immunodeficiency virus type 1 seroconverters. , 2011, Archives of internal medicine.

[24]  James M Robins,et al.  The effect of combined antiretroviral therapy on the overall mortality of HIV-infected individuals , 2010, AIDS.

[25]  J. Robins,et al.  Intervening on risk factors for coronary heart disease: an application of the parametric g-formula. , 2009, International journal of epidemiology.

[26]  S. Cole,et al.  Generalizing evidence from randomized clinical trials to target populations: The ACTG 320 trial. , 2010, American journal of epidemiology.

[27]  P. Flandre.  Atazanavir Plus Ritonavir or Efavirenz as Part of a 3-Drug Regimen for Initial Treatment of HIV-1 , 2011, Annals of Internal Medicine.

[28]  W. Ghali,et al.  Meta-analytic comparison of randomized and nonrandomized studies of breast cancer surgery. , 2012, Canadian journal of surgery. Journal canadien de chirurgie.

[29]  A. Hartz,et al.  A comparison of observational studies and randomized, controlled trials , 2000, American journal of ophthalmology.

[30]  James M Robins,et al.  Per-Protocol Analyses of Pragmatic Trials. , 2017, The New England journal of medicine.

[31]  Vivian C. Wong,et al.  Designs of Empirical Evaluations of Nonexperimental Methods in Field Settings , 2018, Evaluation review.

[32]  L. Bero,et al.  Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials. , 2014, The Cochrane database of systematic reviews.

[33]  J. Robins.  A new approach to causal inference in mortality studies with a sustained exposure period—application to control of the healthy worker survivor effect , 1986 .

[34]  D. Katzenstein,et al.  Atazanavir Plus Ritonavir or Efavirenz as Part of a 3-Drug Regimen for Initial Treatment of HIV-1 , 2011, Annals of Internal Medicine.

[35]  Problems of stopping trials early , 2012, BMJ : British Medical Journal.

[36]  Frank E. Harrell,et al.  Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis , 2001 .

[37]  Jay S. Kaufman,et al.  Bounding Causal Effects Under Uncontrolled Confounding Using Counterfactuals , 2005, Epidemiology.

[38]  James M Robins,et al.  When to Initiate Combined Antiretroviral Therapy to Reduce Mortality and AIDS-Defining Illness in HIV-Infected Persons in Developed Countries , 2011, Annals of Internal Medicine.

[39]  Tyler J. VanderWeele,et al.  Sensitivity Analysis in Observational Research: Introducing the E-Value , 2017, Annals of Internal Medicine.

[40]  James M. Robins,et al.  Comparative Effectiveness of Dynamic Treatment Regimes: An Application of the Parametric G-Formula , 2011, Statistics in biosciences.

[41]  James M. Robins,et al.  Observational Studies Analyzed Like Randomized Experiments: An Application to Postmenopausal Hormone Therapy and Coronary Heart Disease , 2008, Epidemiology.

[42]  Robert Platt,et al.  Specifying a target trial prevents immortal time bias and other self-inflicted injuries in observational analyses. , 2016, Journal of clinical epidemiology.

[43]  R. Salamon,et al.  A Trial of Early Antiretrovirals and Isoniazid Preventive Therapy in Africa. , 2015, The New England journal of medicine.