Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both blinded and non-blinded outcome assessors

Objective To evaluate the impact of non-blinded outcome assessment on estimated treatment effects in randomised clinical trials with binary outcomes. Design Systematic review of trials with both blinded and non-blinded assessment of the same binary outcome. For each trial we calculated the ratio of the odds ratios—the odds ratio from non-blinded assessments relative to the corresponding odds ratio from blinded assessments. A ratio of odds ratios <1 indicated that non-blinded assessors generated more optimistic effect estimates than blinded assessors. We pooled the individual ratios of odds ratios with inverse variance random effects meta-analysis and explored reasons for variation in ratios of odds ratios with meta-regression. We also analysed rates of agreement between blinded and non-blinded assessors and calculated the number of patients needed to be reclassified to neutralise any bias. Data Sources PubMed, Embase, PsycINFO, CINAHL, Cochrane Central Register of Controlled Trials, HighWire Press, and Google Scholar. Eligibility criteria for selecting studies Randomised clinical trials with blinded and non-blinded assessment of the same binary outcome. Results We included 21 trials in the main analysis (with 4391 patients); eight trials provided individual patient data. Outcomes in most trials were subjective—for example, qualitative assessment of the patient’s function. The ratio of the odds ratios ranged from 0.02 to 14.4. The pooled ratio of odds ratios was 0.64 (95% confidence interval 0.43 to 0.96), indicating an average exaggeration of the non-blinded odds ratio by 36%. We found no significant association between low ratios of odds ratios and scores for outcome subjectivity (P=0.27); non-blinded assessor’s overall involvement in the trial (P=0.60); or outcome vulnerability to non-blinded patients (P=0.52). Blinded and non-blinded assessors agreed in a median of 78% of assessments (interquartile range 64-90%) in the 12 trials with available data. The exaggeration of treatment effects associated with non-blinded assessors was induced by the misclassification of a median of 3% of the assessed patients per trial (1-7%). Conclusions On average, non-blinded assessors of subjective binary outcomes generated substantially biased effect estimates in randomised clinical trials, exaggerating odds ratios by 36%. This bias was compatible with a high rate of agreement between blinded and non-blinded outcome assessors and driven by the misclassification of few patients.

[1]  S. Govender,et al.  Recombinant human bone morphogenetic protein-2: a randomized trial in open tibial fractures treated with reamed nail fixation. , 2011, The Journal of bone and joint surgery. American volume.

[2]  D. Burkhoff,et al.  A randomized controlled trial evaluating the safety and efficacy of cardiac contractility modulation in advanced heart failure. , 2011, American heart journal.

[3]  S. Shott,et al.  Vaginal Mesh for Prolapse: A Randomized Controlled Trial , 2010, Obstetrics and gynecology.

[4]  K. Pandher,et al.  Topic of Histopathology Blinding in Nonclinical Safety Biomarker Qualification Studies , 2010, Toxicologic pathology.

[5]  A. Landsman,et al.  Treatment of mild, moderate, and severe onychomycosis using 870- and 930-nm light exposure. , 2010, Journal of the American Podiatric Medical Association.

[6]  D. Altman,et al.  The importance of allocation concealment and patient blinding in osteoarthritis trials: a meta-epidemiologic study. , 2009, Arthritis and rheumatism.

[7]  J. Obeso,et al.  Efficacy and safety of pallidal stimulation in primary dystonia: results of the Spanish multicentric study , 2009, Journal of Neurology, Neurosurgery & Psychiatry.

[8]  S. Yusuf,et al.  Evaluating the benefit of event adjudication of cardiovascular outcomes in large simple RCTs , 2009, Clinical trials.

[9]  E. A. Nelson,et al.  Larval therapy for leg ulcers (VenUS II): randomised controlled trial , 2009, BMJ : British Medical Journal.

[10]  J. Dover,et al.  Review of the Efficacy, Durability, and Safety Data of Two Nonanimal Stabilized Hyaluronic Acid Fillers from a Prospective, Randomized, Comparative, Multicenter Study , 2009, Dermatologic surgery : official publication for American Society for Dermatologic Surgery [et al.].

[11]  G. Guyatt,et al.  Radiographs of hip fractures were digitally altered to mask surgeons to the type of implant without compromising the reliability of quality ratings or making the rating process more difficult. , 2009, Journal of clinical epidemiology.

[12]  Robert Rosenthal,et al.  Meta‐analytically Quantifying the Reliability and Biasability of Forensic Experts , 2008, Journal of forensic sciences.

[13]  G. Guyatt,et al.  Blinding of outcomes in trials of orthopaedic trauma: an opportunity to enhance the validity of clinical trials. , 2008, The Journal of bone and joint surgery. American volume.

[14]  Douglas G Altman,et al.  Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study , 2008, BMJ : British Medical Journal.

[15]  A. Rodgers,et al.  Randomized clinical trial of honey‐impregnated dressings for venous leg ulcers , 2007, The British journal of surgery.

[16]  L. Bass,et al.  A Randomized, Bilateral, Prospective Comparison of Calcium Hydroxylapatite Microspheres versus Human‐Based Collagen for the Correction of Nasolabial Folds , 2007, Dermatologic surgery : official publication for American Society for Dermatologic Surgery [et al.].

[17]  D. Marsh,et al.  Seeing What We Want to See: Confirmation Bias in Animal Behavior Research , 2007 .

[18]  J. Hilden,et al.  Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. , 2007, International journal of epidemiology.

[19]  R. Marti,et al.  Reporting of outcomes in orthopaedic randomized trials: does blinding of outcome assessors matter? , 2007, The Journal of bone and joint surgery. American volume.

[20]  David Moher,et al.  Reporting Methods of Blinding in Randomized Trials Assessing Nonpharmacological Treatments , 2007, PLoS medicine.

[21]  Jun-Mo Yang,et al.  The combined use of needle with hair transplanter for hair recipient sites. , 2007, Dermatologic surgery : official publication for American Society for Dermatologic Surgery [et al.].

[22]  Asbjørn Hróbjartsson,et al.  Who is blinded in randomized clinical trials? A study of 200 trials and a survey of authors , 2006, Clinical trials.

[23]  S. Mirza,et al.  Recombinant human BMP-2 and allograft compared with autogenous bone graft for reconstruction of diaphyseal tibial fractures with cortical defects. A randomized, controlled trial. , 2006, The Journal of bone and joint surgery. American volume.

[24]  M. Swiontkowski,et al.  Recombinant human bone morphogenetic protein-2 in open tibial fractures. A subgroup analysis of data combined from two prospective randomized studies. , 2006, The Journal of bone and joint surgery. American volume.

[25]  M. Feldstein,et al.  Evaluation of a Novel Technique for Wound Closure Using a Barbed Suture , 2006, Plastic and reconstructive surgery.

[26]  R. Topolski,et al.  Clinical and immunological comparison of smallpox vaccine administered to the outer versus the inner upper arms of vaccinia-naive adults. , 2006, Clinical infectious diseases : an official publication of the Infectious Diseases Society of America.

[27]  L. Russell,et al.  Evaluation of a wound dressing using different research methods. , 2004, British journal of nursing.

[28]  L. Russell,et al.  A randomised controlled trial comparing Drawtex with standard dressings for exuding wounds. , 2004, Journal of wound care.

[29]  Y. Mintz,et al.  Effects of intravenous fluid restriction on postoperative complications: comparison of two perioperative fluid regimens: a randomized assessor-blinded multicenter trial. , 2003, Annals of surgery.

[30]  R. Evans,et al.  Extracorporeal Shock Wave Therapy for Treatment of Navicular Syndrome , 2004 .

[31]  F. Pott,et al.  Effects of Intravenous Fluid Restriction on Postoperative Complications: Comparison of Two Perioperative Fluid Regimens: A Randomized Assessor-Blinded Multicenter Trial , 2003, Annals of surgery.

[32]  J. Still,et al.  The use of a collagen sponge/living cell composite material to treat donor sites in burn patients. , 2003, Burns : journal of the International Society for Burn Injuries.

[33]  D. Steward,et al.  The Clinical Effects of Hyaluronic Acid Ester Nasal Dressing (Merogel) on Intranasal Wound Healing after Functional Endoscopic Sinus Surgery , 2003, Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery.

[34]  Alan I Green,et al.  Clozapine treatment for suicidality in schizophrenia: International Suicide Prevention Trial (InterSePT). , 2003, Archives of general psychiatry.

[35]  S. Santavirta,et al.  Recombinant Human Bone Morphogenetic Protein-2 for Treatment of Open Tibial Fractures: A Prospective, Controlled, Randomized Study of Four Hundred and Fifty Patients , 2002, The Journal of bone and joint surgery. American volume.

[36]  H. Höfler,et al.  You get what you expect? A critical appraisal of imaging methodology in endosonographic cancer staging , 2002, Gut.

[37]  Daniel F. Martin,et al.  A controlled trial of valganciclovir as induction therapy for cytomegalovirus retinitis. , 2002, The New England journal of medicine.

[38]  D. Risucci,et al.  The role of blinded interviews in the assessment of surgical residency candidates. , 2001, American journal of surgery.

[39]  G. Guyatt,et al.  Physician interpretations and textbook definitions of blinding terminology in randomized controlled trials. , 2001, JAMA.

[40]  A. Yeung,et al.  Percutaneous transmyocardial laser revascularisation for severe angina: the PACIFIC randomised trial , 2000, The Lancet.

[41]  Daniel Burkhoff,et al.  Transmyocardial laser revascularisation compared with continued medical therapy for treatment of refractory angina pectoris: a prospective randomised trial , 1999, The Lancet.

[42]  R. Simes,et al.  Publication bias: evidence of delayed publication in a cohort study of clinical research projects , 1997, BMJ.

[43]  S. Danner,et al.  Cytomegalovirus retinitis in AIDS patients: a comparative study of intravenous and oral ganciclovir as maintenance therapy. , 1996, AIDS.

[44]  J. Noseworthy,et al.  The impact of blinding on the results of a randomized, placebo‐controlled multiple sclerosis clinical trial , 1994, Neurology.

[45]  P. Mason,et al.  A confocal laser microscopic study of enkephalin-immunoreactive appositions onto physiologically identified neurons in the rostral ventromedial medulla , 1992, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[46]  M. Gent,et al.  Cyclophosphamide and plasma exchange in multiple sclerosis , 1991, The Lancet.

[47]  H. Weiner,et al.  Cyclophosphamide and plasma exchange in multiple sclerosis , 1991, The Lancet.

[48]  N. Laird,et al.  Meta-analysis in clinical trials. , 1986, Controlled clinical trials.

[49]  L. Serbin,et al.  Observer bias in scoring boys' and girls' aggression , 1986 .

[50]  J. Salvia,et al.  Observer Bias: A Methodological Consideration in Special Education Research , 1980 .

[51]  Robert Rosenthal,et al.  How often are our numbers wrong , 1978 .

[52]  J. Titchener.  Experimenter Effects in Behavioral Research. , 1967 .