Can trial quality be reliably assessed from published reports of cancer trials: evaluation of risk of bias assessments in systematic reviews

Objective To evaluate the reliability of risk of bias assessments based on published trial reports, for determining trial inclusion in meta-analyses. Design Reliability evaluation of risk of bias assessments. Data sources 13 published individual participant data (IPD) meta-analyses in cancer were used to source 95 randomised controlled trials. Review methods Risk of bias was assessed using the Cochrane risk of bias tool (RevMan5.1) and accompanying guidance. Assessments were made for individual risk of bias domains and overall for each trial, using information from either trial reports alone or trial reports with additional information collected for IPD meta-analyses. Percentage agreements were calculated for individual domains and overall (<66%=low, ≥66%=fair, ≥90%=good). The two approaches were considered similarly reliable only when agreement was good. Results Percentage agreement between the two methods for sequence generation and incomplete outcome data was fair (69.5% (95% confidence interval 60.2% to 78.7%) and 80.0% (72.0% to 88.0%), respectively). However, percentage agreement was low for allocation concealment, selective outcome reporting, and overall risk of bias (48.4% (38.4% to 58.5%), 42.1% (32.2% to 52.0%), and 54.7% (44.7% to 64.7%), respectively). Supplementary information reduced the proportion of unclear assessments for all individual domains, consequently increasing the number of trials assessed as low risk of bias (and therefore available for inclusion in meta-analyses) from 23 (23%) based on publications alone to 66 (66%) based on publications with additional information. Conclusions Using cancer trial publications alone to assess risk of bias could be unreliable; thus, reviewers should be cautious about using them as a basis for trial inclusion, particularly for those trials assessed as unclear risk. Supplementary information from trialists should be sought to enable appropriate assessments and potentially reduce or overcome some risks of bias. Furthermore, guidance should ensure clarity on what constitutes risk of bias, particularly for the more subjective domains.

[1]  Jayne Tierney,et al.  Chemotherapy in addition to supportive care improves survival in advanced non-small-cell lung cancer: a systematic review and meta-analysis of individual patient data from 16 randomized controlled trials. , 2008, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[2]  M. Parmar,et al.  Adjuvant chemotherapy in invasive bladder cancer: a systematic review and meta-analysis of individual patient data Advanced Bladder Cancer (ABC) Meta-analysis Collaboration. , 2005, European urology.

[3]  C. Vale.  Neoadjuvant chemotherapy in invasive bladder cancer: a systematic review and meta-analysis , 2003, The Lancet.

[4]  David R. Jones,et al.  Empirical assessment of effect of publication bias on meta-analyses , 2000, BMJ : British Medical Journal.

[5]  S. Burdett,et al.  Postoperative radiotherapy in non-small-cell lung cancer: systematic review and meta-analysis of individual patient data from nine randomised controlled trials , 1998, The Lancet.

[6]  D. Moher,et al.  The Revised CONSORT Statement for Reporting Randomized Trials: Explanation and Elaboration , 2001, Annals of Internal Medicine.

[7]  L. Stewart,et al.  To IPD or not to IPD? , 2002, Evaluation & the health professions.

[8]  J. Ioannidis,et al.  The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration , 2009, BMJ : British Medical Journal.

[9]  David J Spiegelhalter,et al.  Bayesian approaches to multiple sources of evidence and uncertainty in complex cost‐effectiveness modelling , 2003, Statistics in medicine.

[10]  Douglas G Altman,et al.  Better reporting of randomised controlled trials: the CONSORT statement , 1996, BMJ.

[11]  R. Sylvester,et al.  Adjuvant chemotherapy for localised resectable soft-tissue sarcoma of adults: meta-analysis of individual data , 1997 .

[12]  J. Minna,et al.  Chemotherapy for non-small cell lung cancer , 1995, BMJ.

[13]  Lisa Hartling,et al.  Applying the Risk of Bias Tool in a Systematic Review of Combination Long-Acting Beta-Agonists and Inhaled Corticosteroids for Persistent Asthma , 2011, PloS one.

[14]  Michele Tarsilla.  Cochrane Handbook for Systematic Reviews of Interventions , 2010, Journal of MultiDisciplinary Evaluation.

[15]  B. Djulbegovic,et al.  Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group , 2004, BMJ : British Medical Journal.

[16]  D. Moher,et al.  CONSORT for Reporting Randomized Controlled Trials in Journal and Conference Abstracts: Explanation and Elaboration , 2008, PLoS medicine.

[17]  L. Stewart,et al.  Adjuvant chemotherapy, with or without postoperative radiotherapy, in operable non-small-cell lung cancer: two meta-analyses of individual patient data , 2010, The Lancet.

[18]  M. Parmar,et al.  Preoperative radiotherapy in esophageal carcinoma: a meta-analysis using individual patient data (Oesophageal Cancer Collaborative Group). , 1998, International journal of radiation oncology, biology, physics.

[19]  Lisa Hartling,et al.  Risk of bias versus quality assessment of randomised controlled trials: cross sectional study , 2009, BMJ : British Medical Journal.

[20]  Adjuvant chemotherapy for localised resectable soft tissue sarcoma in adults. , 2000, The Cochrane database of systematic reviews.

[21]  D. Crowther.  Adjuvant chemotherapy for localised resectable soft-tissue sarcoma of adults: meta-analysis of individual data , 1997, The Lancet.

[22]  B. Djulbegovic,et al.  Published methodological quality of randomized controlled trials does not reflect the actual quality assessed in protocols. , 2012, Journal of clinical epidemiology.

[23]  Keith M Godfrey,et al.  Validation of a maternal questionnaire on correlates of physical activity in preschool children , 2009, The international journal of behavioral nutrition and physical activity.

[24]  J. Ioannidis,et al.  Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias , 2008, PloS one.

[25]  Chemotherapy for high-grade glioma , 2002 .

[26]  L. Stewart,et al.  Chemotherapy for High-Grade Glioma , 2003, Neuroepidemiology.

[27]  R. J. Hayes,et al.  Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. , 1995, JAMA.

[28]  P. Tugwell,et al.  Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? , 2000, The Lancet.

[29]  C. Vale.  Neoadjuvant Chemotherapy in Invasive Bladder Cancer: Update of a Systematic Review and Meta-Analysis of Individual Patient Data: Advanced Bladder Cancer (ABC) Meta-analysis Collaboration , 2005 .

[30]  R. Kolamunnage-Dona,et al.  Assessing the potential for outcome reporting bias in a review: a tutorial , 2010, Trials.

[31]  Parker,et al.  Neoadjuvant chemotherapy for locally advanced cervical cancer: a systematic review and meta-analysis of individual patient data from 21 randomised trials. , 2003, European journal of cancer.

[32]  A R Jadad,et al.  Assessing the quality of reports of randomized clinical trials: is blinding necessary? , 1996, Controlled clinical trials.

[33]  D. Ghersi,et al.  Information on blinding in registered records of clinical trials , 2012, Trials.

[34]  J. Sterne,et al.  The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials , 2011, BMJ : British Medical Journal.

[35]  D. Alberts,et al.  Reducing uncertainties about the effects of chemoradiotherapy for cervical cancer: a systematic review and meta-analysis of individual patient data from 18 randomized trials. , 2008, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[36]  L. Stewart,et al.  Time to publication for results of clinical trials. , 2007, The Cochrane database of systematic reviews.

[37]  O. Brodin,et al.  Chemotherapy in non-small cell lung cancer: a meta-analysis using updated data on individual patients from 52 randomised clinical trials , 1995 .

[38]  J. Sterne,et al.  Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. , 2000, Journal of clinical epidemiology.