Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials.

OBJECTIVE To determine if inadequate approaches to randomized controlled trial design and execution are associated with evidence of bias in estimating treatment effects. DESIGN An observational study in which we assessed the methodological quality of 250 controlled trials from 33 meta-analyses and then analyzed, using multiple logistic regression models, the associations between those assessments and estimated treatment effects. DATA SOURCES Meta-analyses from the Cochrane Pregnancy and Childbirth Database. MAIN OUTCOME MEASURES The associations between estimates of treatment effects and inadequate allocation concealment, exclusions after randomization, and lack of double-blinding. RESULTS Compared with trials in which authors reported adequately concealed treatment allocation, trials in which concealment was either inadequate or unclear (did not report or incompletely reported a concealment approach) yielded larger estimates of treatment effects (P < .001). Odds ratios were exaggerated by 41% for inadequately concealed trials and by 30% for unclearly concealed trials (adjusted for other aspects of quality). Trials in which participants had been excluded after randomization did not yield larger estimates of effects, but that lack of association may be due to incomplete reporting. Trials that were not double-blind also yielded larger estimates of effects (P = .01), with odds ratios being exaggerated by 17%. CONCLUSIONS This study provides empirical evidence that inadequate methodological approaches in controlled trials, particularly those representing poor allocation concealment, are associated with bias. Readers of trial reports should be wary of these pitfalls, and investigators must improve their design, execution, and reporting of trials.

[1]  Curtis L. Meinert,et al.  A proposal for structured reporting of randomized controlled trials. The Standards of Reporting Trials Group. , 1994, JAMA.

[2]  S Greenland,et al.  Invited commentary: a critical look at some popular meta-analytic methods. , 1994, American journal of epidemiology.

[3]  D G Altman,et al.  Assessing the quality of randomization from reports of controlled trials published in obstetrics and gynecology journals. , 1994, JAMA.

[4]  K. Schulz Methodological quality and bias in randomised controlled trials. , 1994 .

[5]  C D Naylor,et al.  Incorporating variations in the quality of individual randomized trials into meta-analysis. , 1992, Journal of clinical epidemiology.

[6]  Douglas G. Altman,et al.  RANDOMISATION : ESSENTIAL FOR REDUCING BIAS , 1991 .

[7]  D. Paintin Effective Care in Pregnancy and Childbirth , 1990, British journal of obstetrics and gynaecology.

[8]  J D Emerson,et al.  An empirical study of the possible relation of treatment differences to quality scores in controlled randomized clinical trials. , 1990, Controlled clinical trials.

[9]  D. G. Altman,et al.  Randomisation and baseline comparisons in clinical trials , 1990, The Lancet.

[10]  G A Colditz,et al.  How study design affects outcomes in comparisons of therapy. II: Surgical. , 1989, Statistics in medicine.

[11]  F. Mosteller,et al.  How study design affects outcomes in comparisons of therapy. I: Medical. , 1989, Statistics in medicine.

[12]  R Peto,et al.  Why do we need systematic overviews of randomized trials? , 1987, Statistics in medicine.

[13]  L Mutch,et al.  The Oxford Database of Perinatal Trials: developing a register of published reports of controlled trials. , 1986, Controlled clinical trials.

[14]  A Liberati,et al.  A quality assessment of randomized control trials of primary treatment of breast cancer. , 1986, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[15]  T C Chalmers,et al.  Bias in treatment assignment in controlled clinical trials. , 1983, The New England journal of medicine.

[16]  T. Cook,et al.  A Meta‐Analytic Analysis of Effects of Psychoeducational Interventions on Length of Postsurgical Hospital Stay , 1983, Nursing research.

[17]  F. Mosteller,et al.  Reporting on methods in clinical trials. , 2019, The New England journal of medicine.

[18]  T C Chalmers,et al.  Randomized versus historical controls for clinical trials. , 1982, The American journal of medicine.

[19]  T C Chalmers,et al.  A method for assessing the quality of a randomized control trial. , 1981, Controlled clinical trials.

[20]  F. Mosteller,et al.  Reporting standards and research strategies for controlled trials , 1980 .

[21]  Pocock Sj,et al.  Allocation of patients to treatment in clinical trials. , 1979 .

[22]  S J Pocock,et al.  Allocation of patients to treatment in clinical trials. , 1979, Biometrics.

[23]  T C Chalmers,et al.  Evidence favoring the use of anticoagulants in the hospital phase of acute myocardial infarction. , 1977, The New England journal of medicine.