How to spot bias and other potential problems in randomised controlled trials

These days, all clinical trials should be reported using the CONSORT guidelines1 (table 1); indeed JNNP recommends this in its instructions for authors. However, not all trials are reported in this way, and many journals do not insist on it. Thus some trials may have been carried out adequately but reported inadequately, while others have been carried out inadequately. Our aim in this article is to guide clinicians in what to look for in a report of a randomised controlled trial (RCT), so they can assess whether the trial was done adequately; we do not intend it to be a guide on how to do an RCT, as there are many such guides available.2 View this table: Table 1 Checklist of items to include when reporting a randomised trial (from the CONSORT statement) The two crucial principles in clinical research are to minimise bias and to increase precision. If a study is not designed with these two principles in mind, no amount of analysis will sort them out. We will discuss some of the major biases to look out for, issues related to precision, and some other aspects of statistical analysis. Bias is any departure of results from the truth. An RCT is less susceptible to bias than other study designs for assessing therapeutic interventions. However, just because a study is randomised does not mean it is unbiased. There are at least seven important potential sources of bias in RCTs, which are discussed below. When assessing bias, it is important to consider its magnitude as well as its direction. Trials that have shown large treatment effects may still be positive after a small bias has been accounted for. ### Poor allocation concealment In a good trial, the treatment allocation is concealed during the randomisation procedure. In other words, at the time a clinician randomises a patient they …

[1]  I. Chalmers,et al.  The Landscape and Lexicon of Blinding in Randomized Trials , 2002, Annals of Internal Medicine.

[2]  T. Peters,et al.  Subgroup analyses in randomised controlled trials: quantifying the risks of false-positives and false-negatives. , 2001, Health technology assessment.

[3]  A randomized trial of aspirin and sulfinpyrazone in threatened stroke. , 1978, The New England journal of medicine.

[4]  G. Guyatt,et al.  Post-randomisation exclusions: the intention to treat principle and excluding patients from analysis , 2002, BMJ : British Medical Journal.

[5]  S. Pocock,et al.  Subgroup analysis, covariate adjustment and baseline comparisons in clinical trial reporting: current practiceand problems , 2002, Statistics in medicine.

[6]  M. Kaste,et al.  Recommendations for the Relationship Between Sponsors and Investigators in the Design and Conduct of Clinical Stroke Trials , 2003, Stroke.

[7]  R. Collins,et al.  Reliable assessment of the effects of treatment on mortality and major morbidity, I: clinical trials , 2001, The Lancet.

[8]  Kenneth F Schulz,et al.  For Personal Use. Only Reproduce with Permission from the Lancet Publishing Group. Exclusions before Randomisation Exclusions after Randomisation Sample Size Slippages in Randomised Trials: Exclusions and the Lost and Wayward , 2022 .

[9]  M. Palmer.  Clinical Trials: A Practical Approach , 1985 .

[10]  Walker,et al.  Collaborative overview of randomised trials of antiplatelet therapy Prevention of death, myocardial infarction, and stroke by prolonged antiplatelet therapy in various categories of patients , 1994 .

[11]  M. Clarke,et al.  DICE 2: A FURTHER INVESTIGATION OF THE EFFECTS OF CHANCE IN LIFE, DEATH AND SUBGROUP ANALYSES , 2001, International journal of clinical practice.

[12]  J P Siegel,et al.  Equivalence and noninferiority trials. , 2000, American heart journal.

[13]  D. Liebeskind,et al.  Trends in Acute Ischemic Stroke Trials Through the 20th Century , 2001, Stroke.

[14]  D Moher,et al.  The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. , 2001, Annals of internal medicine.

[15]  D G Altman,et al.  Statistics notes: Interaction 3: How to examine heterogeneity , 1996 .

[16]  C. Viscoli,et al.  Stratified randomization for clinical trials. , 1999, Journal of clinical epidemiology.

[17]  J. Buring,et al.  Ways of measuring rates of recurrent events , 1996, BMJ.

[18]  D. DeMets,et al.  Surrogate End Points in Clinical Trials: Are We Being Misled? , 1996, Annals of Internal Medicine.

[19]  C. Gross,et al.  Scope and impact of financial conflicts of interest in biomedical research: a systematic review. , 2003, JAMA.

[20]  B. Djulbegovic,et al.  Pharmaceutical industry sponsorship and research outcome and quality: systematic review , 2003, BMJ : British Medical Journal.

[21]  D G Altman,et al.  Concealing treatment allocation in randomised trials , 2001, BMJ : British Medical Journal.

[22]  D. G. Altman,et al.  Randomisation and baseline comparisons in clinical trials , 1990, The Lancet.

[23]  L. Bolognese,et al.  RANDOMISED TRIAL OF INTRAVENOUS STREPTOKINASE, ORAL ASPIRIN, BOTH, OR NEITHER AMONG 17 187 CASES OF SUSPECTED ACUTE MYOCARDIAL INFARCTION: ISIS-2 , 1988, The Lancet.

[24]  J. Lubsen,et al.  Combined endpoints: can we use them? , 2002, Statistics in medicine.

[25]  S. Assmann,et al.  Subgroup analysis and other (mis)uses of baseline data in clinical trials , 2000, The Lancet.

[26]  C. Warlow.  Advanced issues in the design and conduct of randomized clinical trials: the bigger the better? , 2002, Statistics in medicine.

[27]  E. Dooley.  National Institute of Neurological Disorders and Stroke , 2006 .

[28]  D. Moher,et al.  The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials , 2001, The Lancet.

[29]  P. Bath,et al.  Quality of full and final publications reporting acute stroke trials: a systematic review. , 1998, Stroke.

[30]  D. Clayton,et al.  Be skeptical about unexpected large apparent treatment effects: the case of an MRC AML12 randomization. , 2003, Controlled clinical trials.

[31]  Kenneth F Schulz,et al.  Allocation concealment in randomised trials: defending against deciphering , 2002, The Lancet.

[32]  Steff Lewis,et al.  Forest plots: trying to see the wood and the trees , 2001, BMJ : British Medical Journal.

[33]  Joseph P. Broderick,et al.  Tissue plasminogen activator for acute ischemic stroke. The National Institute of Neurological Disorders and Stroke rt-PA Stroke Study Group. , 1995 .

[34]  D. Torgerson,et al.  Randomisation methods in controlled trials , 1998, BMJ.

[35]  Koroshetz Wj,et al.  Tissue plasminogen activator for acute ischemic stroke. , 1996, The New England journal of medicine.

[36]  J. Mann.  Truths about the NINDS study: setting the record straight. , 2002, The Western journal of medicine.

[37]  S. Goodman,et al.  The Use of Predicted Confidence Intervals When Planning Experiments and the Misuse of Power When Interpreting Results , 1994, Annals of Internal Medicine.

[38]  S Senn,et al.  Analysis of serial measurements in medical research. , 1990, BMJ.

[39]  G. Guyatt,et al.  Users' guides to the medical literature: XIX. Applying clinical trial results. A. How to use an article measuring the effect of an intervention on surrogate end points. Evidence-Based Medicine Working Group. , 1999, JAMA.

[40]  J. Noseworthy,et al.  The impact of blinding on the results of a randomized, placebo‐controlled multiple sclerosis clinical trial , 1994, Neurology.

[41]  A. Ebbutt,et al.  Practical issues in equivalence trials. , 1998, Statistics in medicine.

[42]  C. Warlow,et al.  Performance of a Statistical Model to Predict Stroke Outcome in the Context of a Large, Simple, Randomized, Controlled Trial of Feeding , 2003, Stroke.

[43]  J M Bland,et al.  Trials which randomize practices I: how should they be analysed? , 1998, Family practice.

[44]  J A Lewis,et al.  Trials to assess equivalence: the importance of rigorous methods , 1996, BMJ.

[45]  G. Guyatt,et al.  Users' guides to the medical literature. , 1993, JAMA.