Impact of blinding on estimated treatment effects in randomised clinical trials: meta-epidemiological study

Abstract Objectives To study the impact of blinding on estimated treatment effects, and their variation between trials; differentiating between blinding of patients, healthcare providers, and observers; detection bias and performance bias; and types of outcome (the MetaBLIND study). Design Meta-epidemiological study. Data source Cochrane Database of Systematic Reviews (2013-14). Eligibility criteria for selecting studies Meta-analyses with both blinded and non-blinded trials on any topic. Review methods Blinding status was retrieved from trial publications and authors, and results retrieved automatically from the Cochrane Database of Systematic Reviews. Bayesian hierarchical models estimated the average ratio of odds ratios (ROR), and estimated the increases in heterogeneity between trials, for non-blinded trials (or of unclear status) versus blinded trials. Secondary analyses adjusted for adequacy of concealment of allocation, attrition, and trial size, and explored the association between outcome subjectivity (high, moderate, low) and average bias. An ROR lower than 1 indicated exaggerated effect estimates in trials without blinding. Results The study included 142 meta-analyses (1153 trials). The ROR for lack of blinding of patients was 0.91 (95% credible interval 0.61 to 1.34) in 18 meta-analyses with patient reported outcomes, and 0.98 (0.69 to 1.39) in 14 meta-analyses with outcomes reported by blinded observers. The ROR for lack of blinding of healthcare providers was 1.01 (0.84 to 1.19) in 29 meta-analyses with healthcare provider decision outcomes (eg, readmissions), and 0.97 (0.64 to 1.45) in 13 meta-analyses with outcomes reported by blinded patients or observers. The ROR for lack of blinding of observers was 1.01 (0.86 to 1.18) in 46 meta-analyses with subjective observer reported outcomes, with no clear impact of degree of subjectivity. Information was insufficient to determine whether lack of blinding was associated with increased heterogeneity between trials. The ROR for trials not reported as double blind versus those that were double blind was 1.02 (0.90 to 1.13) in 74 meta-analyses. Conclusion No evidence was found for an average difference in estimated treatment effect between trials with and without blinded patients, healthcare providers, or outcome assessors. These results could reflect that blinding is less important than often believed or meta-epidemiological study limitations, such as residual confounding or imprecision. At this stage, replication of this study is suggested and blinding should remain a methodological safeguard in trials.

[1]  J. Sterne,et al.  Ten questions to consider when interpreting results of a meta‐epidemiological study—the MetaBLIND study as a case , 2019, Research synthesis methods.

[2]  Natalie S Blencowe,et al.  RoB 2: a revised tool for assessing risk of bias in randomised trials , 2019, BMJ.

[3]  K. Ashkan,et al.  Patient-Reported Outcome Measures in Neurosurgery: A Review of the Current Literature , 2018, Neurosurgery.

[4]  H. Saltaji,et al.  Influence of blinding on treatment effect size estimate in randomized controlled trials of oral health interventions , 2018, BMC Medical Research Methodology.

[5]  Julian P T Higgins,et al.  Association Between Risk-of-Bias Assessments and Results of Randomized Trials in Cochrane Reviews: The ROBES Meta-Epidemiologic Study , 2017, American journal of epidemiology.

[6]  J. Higgins,et al.  Label‐invariant models for the analysis of meta‐epidemiological data , 2017, Statistics in medicine.

[7]  H. Saltaji,et al.  Blinding in Physical Therapy Trials and Its Association with Treatment Effects: A Meta-epidemiological Study , 2017, American journal of physical medicine & rehabilitation.

[8]  A. Hrõbjartsson,et al.  Unreported formal assessment of unblinding occurred in 4 of 10 randomized clinical trials, unreported loss of blinding in 1 of 10 trials. , 2017, Journal of clinical epidemiology.

[9]  L. Trinquart,et al.  Empirical evaluation of which trial characteristics are associated with treatment effect estimates. , 2016, Journal of clinical epidemiology.

[10]  Jelena Savović,et al.  Empirical Evidence of Study Design Biases in Randomized Trials: Systematic Review of Meta-Epidemiological Studies , 2016, PloS one.

[11]  Julian P T Higgins,et al.  Sample size calculation for meta‐epidemiological studies , 2016, Statistics in medicine.

[12]  Jos J. Mellema,et al.  Influence of Priming on Patient-Reported Outcome Measures: A Randomized Controlled Trial. , 2016, Psychosomatics.

[13]  A. Hrõbjartsson,et al.  Subjective and objective outcomes in randomized clinical trials: definitions differed in methods publications and were often absent from trial reports. , 2014, Journal of clinical epidemiology.

[14]  R. Golub,et al.  Meta-analysis as evidence: building a better pyramid. , 2014, JAMA.

[15]  Ann Sofia Skou Thomsen,et al.  Bias due to lack of patient blinding in clinical trials. A systematic review of trials randomizing patients to blind and nonblind sub-studies. , 2014, International journal of epidemiology.

[16]  J. Hilden,et al.  Observer bias in randomized clinical trials with time-to-event outcomes: systematic review of trials with both blinded and non-blinded outcome assessors. , 2014, International journal of epidemiology.

[17]  J. Hilden,et al.  Observer bias in randomized clinical trials with measurement scale outcomes: a systematic review of trials with both blinded and nonblinded assessors , 2013, Canadian Medical Association Journal.

[18]  Jonathan J Deeks,et al.  Issues relating to study design and risk of bias when including non‐randomized studies in systematic reviews on the effects of interventions , 2013, Research synthesis methods.

[19]  Ethan M Balk,et al.  Influence of Reported Study Design Characteristics on Intervention Effect Estimates From Randomized, Controlled Trials , 2012, Annals of Internal Medicine.

[20]  G. Guyatt,et al.  Specific instructions for estimating unclearly reported blinding status in randomized trials were reliable and valid. , 2012, Journal of clinical epidemiology.

[21]  Isabelle Boutron,et al.  Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both blinded and non-blinded outcome assessors , 2012, BMJ : British Medical Journal.

[22]  J. Sterne,et al.  The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials , 2011, BMJ : British Medical Journal.

[23]  J. Ware,et al.  Pragmatic trials--guides to better patient care? , 2011, The New England journal of medicine.

[24]  C. Bias.  The Cochrane Collaboration's tool for assessing risk of bias in randomised trials , 2011 .

[25]  D. Altman,et al.  The importance of allocation concealment and patient blinding in osteoarthritis trials: a meta-epidemiologic study. , 2009, Arthritis and rheumatism.

[26]  Douglas G. Altman,et al.  Models for potentially biased evidence in meta‐analysis using empirically based priors , 2009 .

[27]  Asbjørn Hróbjartsson,et al.  Who is blinded in randomized clinical trials? A study of 200 trials and a survey of authors , 2006, Clinical trials.

[28]  David R. Jones,et al.  How vague is vague? A simulation study of the impact of the use of vague prior distributions in MCMC using WinBUGS , 2005, Statistics in medicine.

[29]  Douglas G Altman,et al.  Epidemiology and reporting of randomised trials published in PubMed journals , 2005, The Lancet.

[30]  Douglas G Altman,et al.  Statistical methods for assessing the influence of study characteristics on treatment effects in ‘meta‐epidemiological’ research , 2002, Statistics in medicine.

[31]  G. Guyatt,et al.  Physician interpretations and textbook definitions of blinding terminology in randomized controlled trials. , 2001, JAMA.

[32]  S. Chinn.  A simple method for converting an odds ratio to effect size for use in meta-analysis. , 2000, Statistics in medicine.

[33]  Andrew Thomas,et al.  WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility , 2000, Stat. Comput..

[34]  T. Kaptchuk,et al.  Intentional Ignorance: A History of Blind Assessment and Placebo Controls in Medicine , 1998, Bulletin of the history of medicine.

[35]  R. Nickerson.  Confirmation Bias: A Ubiquitous Phenomenon in Many Guises , 1998 .

[36]  R. J. Hayes,et al.  Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. , 1995, JAMA.

[37]  R. Rosenthal,et al.  On the social psychology of the psychological experiment: the experimenter's hypothesis as unintended determinant of experimental results. , 1963, American scientist.