Ten questions to consider when interpreting results of a meta‐epidemiological study—the MetaBLIND study as a case

Randomized clinical trials underpin evidence‐based clinical practice, but flaws in their conduct may lead to biased estimates of intervention effects and hence invalid treatment recommendations. The main approach to the empirical study of bias is to collate a number of meta‐analyses and, within each, compare the results of trials with and without a methodological characteristic such as blinding of participants and health professionals. Estimated within‐meta‐analysis differences are combined across meta‐analyses, leading to an estimate of mean bias. Such “meta‐epidemiological” studies are published in increasing numbers and have the potential to inform trial design, assessment of risk of bias, and reporting guidelines. However, their interpretation is complicated by issues of confounding, imprecision, and applicability. We developed a guide for interpreting meta‐epidemiological studies, illustrated using MetaBLIND, a large study on the impact of blinding. Applying generally accepted principles of research methodology to meta‐epidemiology, we framed 10 questions covering the main issues to consider when interpreting results of such studies, including risk of systematic error, risk of random error, issues related to heterogeneity, and theoretical plausibility. We suggest that readers of a meta‐epidemiological study reflect comprehensively on the research question posed in the study, whether an experimental intervention was unequivocally identified for all included trials, the risk of misclassification of the trial characteristic, and the risk of confounding, i.e the adequacy of any adjustment for the likely confounders. We hope that our guide to interpretation of results of meta‐epidemiological studies is helpful for readers of such studies.

[1]  J. Sterne,et al.  Impact of blinding on estimated treatment effects in randomised clinical trials: meta-epidemiological study , 2020, BMJ.

[2]  J. Higgins,et al.  Cochrane Handbook for Systematic Reviews of Interventions , 2010, International Coaching Psychology Review.

[3]  J. Sterne Commentary: does the selective inversion approach demonstrate bias in the results of studies using routinely collected data? , 2018, British Medical Journal.

[4]  M. Page,et al.  Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review , 2018, BMJ Open.

[5]  P. Tugwell,et al.  Some Cochrane risk-of-bias items are not important in osteoarthritis trials: a meta-epidemiological study based on Cochrane reviews. , 2017, Journal of clinical epidemiology.

[6]  Julian P T Higgins,et al.  Association Between Risk-of-Bias Assessments and Results of Randomized Trials in Cochrane Reviews: The ROBES Meta-Epidemiologic Study , 2017, American journal of epidemiology.

[7]  Zhen Wang,et al.  Guidelines for reporting meta-epidemiological methodology research , 2017, Evidence-Based Medicine.

[8]  J. Higgins,et al.  Label‐invariant models for the analysis of meta‐epidemiological data , 2017, Statistics in medicine.

[9]  L. Borges,et al.  Understanding Meta-Epidemiological Studies , 2016 .

[10]  Jelena Savović,et al.  Empirical Evidence of Study Design Biases in Randomized Trials: Systematic Review of Meta-Epidemiological Studies , 2016, PloS one.

[11]  Julian P T Higgins,et al.  Sample size calculation for meta‐epidemiological studies , 2016, Statistics in medicine.

[12]  Asbjørn Hróbjartsson,et al.  Lack of blinding of outcome assessors in animal model experiments implies risk of observer bias. , 2014, Journal of clinical epidemiology.

[13]  Ann Sofia Skou Thomsen,et al.  Bias due to lack of patient blinding in clinical trials. A systematic review of trials randomizing patients to blind and nonblind sub-studies. , 2014, International journal of epidemiology.

[14]  J. Hilden,et al.  Observer bias in randomized clinical trials with time-to-event outcomes: systematic review of trials with both blinded and non-blinded outcome assessors. , 2014, International journal of epidemiology.

[15]  G. Gyte,et al.  Restricting oral fluid and food intake during labour. , 2013, The Cochrane database of systematic reviews.

[16]  L. Trinquart,et al.  Influence of trial sample size on treatment effect estimates: meta-epidemiological study , 2013, BMJ : British Medical Journal.

[17]  J. Hilden,et al.  Observer bias in randomized clinical trials with measurement scale outcomes: a systematic review of trials with both blinded and nonblinded assessors , 2013, Canadian Medical Association Journal.

[18]  Ross J. Harris,et al.  Influence of reported study design characteristics on intervention effect estimates from randomised controlled trials: combined analysis of meta-epidemiological studies. , 2012, Health technology assessment.

[19]  Ethan M Balk,et al.  Influence of Reported Study Design Characteristics on Intervention Effect Estimates From Randomized, Controlled Trials , 2012, Annals of Internal Medicine.

[20]  G. Guyatt,et al.  Specific instructions for estimating unclearly reported blinding status in randomized trials were reliable and valid. , 2012, Journal of clinical epidemiology.

[21]  Isabelle Boutron,et al.  Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both blinded and non-blinded outcome assessors , 2012, BMJ : British Medical Journal.

[22]  Isabelle Boutron,et al.  Impact of single centre status on estimates of intervention effects in trials with continuous outcomes: meta-epidemiological study , 2012, BMJ : British Medical Journal.

[23]  I. Boutron,et al.  Blinding in Randomized Clinical Trials: Imposed Impartiality , 2011, Clinical pharmacology and therapeutics.

[24]  Isabelle Boutron,et al.  Single-Center Trials Show Larger Treatment Effects Than Multicenter Trials: Evidence From a Meta-epidemiologic Study , 2011, Annals of Internal Medicine.

[25]  K. Dickersin,et al.  Metabias: A Challenge for Comparative Effectiveness Research , 2011, Annals of Internal Medicine.

[26]  Michele Tarsilla Cochrane Handbook for Systematic Reviews of Interventions , 2010, Journal of MultiDisciplinary Evaluation.

[27]  D. Altman,et al.  The importance of allocation concealment and patient blinding in osteoarthritis trials: a meta-epidemiologic study. , 2009, Arthritis and rheumatism.

[28]  P. Seed,et al.  Effect of food intake during labour on obstetric outcome: randomised controlled trial , 2009, BMJ : British Medical Journal.

[29]  Douglas G. Altman,et al.  Models for potentially biased evidence in meta‐analysis using empirically based priors , 2009 .

[30]  J. Hilden,et al.  Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. , 2007, International journal of epidemiology.

[31]  Asbjørn Hróbjartsson,et al.  Who is blinded in randomized clinical trials? A study of 200 trials and a survey of authors , 2006, Clinical trials.

[32]  E. Hodnett,et al.  The effect of unrestricted oral carbohydrate intake on labor progress. , 2005, Journal of obstetric, gynecologic, and neonatal nursing : JOGNN.

[33]  Gordon H Guyatt,et al.  An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods. , 2004, Journal of clinical epidemiology.

[34]  Douglas G Altman,et al.  Statistical methods for assessing the influence of study characteristics on treatment effects in ‘meta‐epidemiological’ research , 2002, Statistics in medicine.

[35]  Ethan M Balk,et al.  Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. , 2002, JAMA.

[36]  S. Cessie,et al.  A double‐blind, randomised, placebo controlled study on the influence of carbohydrate solution intake during labour , 2002, BJOG : an international journal of obstetrics and gynaecology.

[37]  P. Seed,et al.  An evaluation of isotonic "sport drinks" during labor. , 2002, Anesthesia and analgesia.

[38]  Christian Gluud,et al.  Reported Methodologic Quality and Discrepancies between Large and Small Randomized Trials in Meta-Analyses , 2001, Annals of Internal Medicine.

[39]  G. Guyatt,et al.  Physician interpretations and textbook definitions of blinding terminology in randomized controlled trials. , 2001, JAMA.

[40]  P. Seed,et al.  Eating in labour. A randomised controlled trial assessing the risks and benefits , 1999, Anaesthesia.

[41]  D. Cook,et al.  Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? , 1998, The Lancet.

[42]  D J Torgerson,et al.  Pragmatic trials: lab meets bedside , 2019, The British journal of dermatology.

[43]  R. J. Hayes,et al.  Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. , 1995, JAMA.

[44]  J D Emerson,et al.  An empirical study of the possible relation of treatment differences to quality scores in controlled randomized clinical trials. , 1990, Controlled clinical trials.

[45]  A. B. Hill The Environment and Disease: Association or Causation? , 1965, Proceedings of the Royal Society of Medicine.

[46]  Ross J. Harris,et al.  Correction: reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. , 2008, Annals of internal medicine.

[47]  J. Sterne,et al.  How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. , 2003, Health technology assessment.

[48]  M. Dufour,et al.  Epidemiology. Introduction. , 1995, Recent developments in alcoholism : an official publication of the American Medical Society on Alcoholism, the Research Society on Alcoholism, and the National Council on Alcoholism.