Systematic Differences in Effect Estimates Between Observational Studies and Randomized Control Trials in Meta-Analyses Combining Both Study Designs in Nephrology

The limited availability of randomized controlled trials (RCTs) in nephrology undermines causal inferences in meta-analyses. Systematic reviews of observational studies have grown more common under such circumstances. We conducted systematic reviews of all comparative observational studies in nephrology from 2006 to 2016 to assess the trends in the past decade. We then focused on the meta-analyses combining observational studies and RCTs to evaluate the systematic differences in effect estimates between study designs using two statistical methods: by estimating the ratio of odds ratios (ROR) of the pooled OR obtained from observational studies versus those from RCTs and by examining the discrepancies in their statistical significance. The number of systematic reviews of observational studies in nephrology had grown by 11.7-fold in the past decade. Among 56 records combining observational studies and RCTs, ROR suggested that the estimates between study designs agreed well (ROR: 1.05, 95% confidence interval: 0.90-1.23). However, almost half of the reviews led to discrepant interpretations in terms of statistical significance. In conclusion, the findings based on ROR might encourage researchers to justify the inclusion of observational studies in meta-analyses. However, caution is needed as the interpretations based on statistical significance were less concordant than those based on ROR.

[1]  D. Zurakowski,et al.  Five Steps to Successfully Implement and Evaluate Propensity Score Matching in Clinical Research Studies , 2018, Anesthesia and analgesia.

[2]  K. Huybrechts,et al.  Implications of the Propensity Score Matching Paradox in Pharmacoepidemiology , 2018, American journal of epidemiology.

[3]  P. Tugwell,et al.  AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both , 2017, British Medical Journal.

[4]  R. Lewis,et al.  Confounding by Indication in Clinical Research. , 2016, JAMA.

[5]  M. Hernán,et al.  ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions , 2016, British Medical Journal.

[6]  M. A. Choudhury,et al.  A Methodological Overview , 2016 .

[7]  T. Furukawa,et al.  Replication and contradiction of highly cited research papers in psychiatry: 10-year follow-up , 2015, British Journal of Psychiatry.

[8]  John P A Ioannidis,et al.  Assessment of vibration of effects due to model specification can demonstrate the instability of observational associations. , 2015, Journal of clinical epidemiology.

[9]  L. Bero,et al.  Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials. , 2014, The Cochrane database of systematic reviews.

[10]  L. Trinquart,et al.  Comparison of Treatment Effect Estimates From Prospective Nonrandomized Studies With Propensity Score Analysis and Randomized Controlled Trials of Surgical Procedures , 2014, Annals of surgery.

[11]  P. Tugwell,et al.  An introduction to methodological issues when including non‐randomised studies in systematic reviews on the effects of interventions , 2013, Research synthesis methods.

[12]  H. Schünemann,et al.  [GRADE guidelines: 9. Rating up the quality of evidence]. , 2013, Zeitschrift fur Evidenz, Fortbildung und Qualitat im Gesundheitswesen.

[13]  Nancy D Berkman,et al.  Development of the RTI item bank on risk of bias and precision of observational studies. , 2012, Journal of clinical epidemiology.

[14]  D. Molony,et al.  Randomized controlled trials in nephrology: state of the evidence and critiquing the evidence. , 2012, Advances in chronic kidney disease.

[15]  John P A Ioannidis,et al.  Prognostic effect size of cardiovascular biomarkers in datasets from observational studies versus randomised trials: meta-epidemiology study , 2011, BMJ : British Medical Journal.

[16]  P. Shekelle,et al.  Observational studies in systematic [corrected] reviews of comparative effectiveness: AHRQ and the Effective Health Care Program. , 2011, Journal of clinical epidemiology.

[17]  O. Kuss,et al.  Treatments effects from randomized trials and propensity score analyses were similar in similar populations in an example from cardiac surgery. , 2011, Journal of clinical epidemiology.

[18]  S. Palmer,et al.  Trial quality in nephrology: how are we measuring up? , 2011, American journal of kidney diseases : the official journal of the National Kidney Foundation.

[19]  C. Schmid,et al.  Loss to analysis in randomized controlled trials in CKD. , 2011, American journal of kidney diseases : the official journal of the National Kidney Foundation.

[20]  Su Golder,et al.  Meta-analyses of Adverse Effects Data Derived from Randomised Controlled Trials as Compared to Observational Studies: Methodological Overview , 2011, PLoS medicine.

[21]  J. Higgins.  Cochrane handbook for systematic reviews of interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration , 2011 .

[22]  T. Greene.  Randomized and observational studies in nephrology: how strong is the evidence? , 2009, American journal of kidney diseases : the official journal of the National Kidney Foundation.

[23]  Amit X Garg,et al.  Systematic review and meta-analysis: when one study is just not enough. , 2008, Clinical journal of the American Society of Nephrology : CJASN.

[24]  P. Rochon,et al.  Readers should systematically assess methods used to identify, measure and analyze confounding in observational cohort studies. , 2007, Journal of clinical epidemiology.

[25]  D. Charytan,et al.  The exclusion of patients with chronic kidney disease from clinical trials in coronary artery disease. , 2006, Kidney international.

[26]  Jill Hayden,et al.  Evaluation of the Quality of Prognosis Studies in Systematic Reviews , 2006, Annals of Internal Medicine.

[27]  Jonathan C Craig,et al.  The number, quality, and coverage of randomized controlled trials in nephrology. , 2004, Journal of the American Society of Nephrology : JASN.

[28]  Douglas G Altman,et al.  Statistical methods for assessing the influence of study characteristics on treatment effects in ‘meta‐epidemiological’ research , 2002, Statistics in medicine.

[29]  J. Ray.  Evidence in upheaval: incorporating observational data into clinical practice. , 2002, Archives of internal medicine.

[30]  Evidence-based medicine in nephrology: identifying and critically appraising the literature. , 2000, Nephrology, dialysis, transplantation : official publication of the European Dialysis and Transplant Association - European Renal Association.

[31]  S. Barton,et al.  Which clinical studies provide the best evidence? , 2000, BMJ : British Medical Journal.

[32]  S. Rosenzweig,et al.  Ethics of clinical trials. , 2000, The New England journal of medicine.

[33]  N. Black,et al.  The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. , 1998, Journal of epidemiology and community health.

[34]  N. Black.  Why we need observational studies to evaluate the effectiveness of health care , 1996, BMJ.

[35]  D. Sackett,et al.  Cochrane Collaboration , 1994, BMJ.

[36]  Bernardo Kucinski.  9. , 1794, The Past is an Imperfect Tense.