Bmc Medical Research Methodology Open Access Recommendations by Cochrane Review Groups for Assessment of the Risk of Bias in Studies

BackgroundAssessing the risk of bias in individual studies in a systematic review can be done using individual components or by summarizing the study quality in an overall score.MethodsWe examined the instructions to authors of the 50 Cochrane Review Groups that focus on clinical interventions for recommendations on methodological quality assessment of studies.ResultsForty-one of the review groups (82%) recommended quality assessment using components and nine using a scale. All groups recommending components recommended to assess concealment of allocation, compared to only two of the groups recommending scales (P < 0.0001). Thirty-five groups (70%) recommended assessment of sequence generation and 21 groups (42%) recommended assessment of intention-to-treat analysis. Only 28 groups (56%) had specific recommendations for using the quality assessment of studies analytically in reviews, with sensitivity analysis, quality as an inclusion threshold and subgroup analysis being the most commonly recommended methods. The scales recommended had problems in the individual items and some of the groups recommending components recommended items not related to bias in their quality assessment.ConclusionWe found that recommendations by some groups were not based on empirical evidence and many groups had no recommendations on how to use the quality assessment in reviews. We suggest that all Cochrane Review Groups refer to the Cochrane Handbook for Systematic Reviews of Interventions, which is evidence-based, in their instructions to authors and that their own guidelines are kept to a minimum and describe only how methodological topics that are specific to their fields should be handled.

[1]  D G Altman,et al.  Assessing the quality of randomization from reports of controlled trials published in obstetrics and gynecology journals. , 1994, JAMA.

[2]  A Laupacis,et al.  Assessing the quality of randomized trials: reliability of the Jadad scale. , 1999, Controlled clinical trials.

[3]  A. Hrõbjartsson,et al.  Somatostatin analogues for acute bleeding oesophageal varices. , 2008, The Cochrane database of systematic reviews.

[4]  D. Machin,et al.  Intention to treat--who should use ITT? , 1993, British Journal of Cancer.

[5]  A D Oxman,et al.  Randomisation to protect against selection bias in healthcare trials. , 2011, The Cochrane database of systematic reviews.

[6]  J. Sterne,et al.  How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. , 2003, Health technology assessment.

[7]  J. Deeks.  Systematic reviews evaluating effects of health care interventions : issues of synthesis and bias , 2007 .

[8]  Lesley A Stewart,et al.  Investigating patient exclusion bias in meta-analysis. , 2004, International journal of epidemiology.

[9]  A. Bostom,et al.  Postmenopausal estrogen therapy and cardiovascular disease. , 1992, The New England journal of medicine.

[10]  A R Jadad,et al.  Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. , 1995, Controlled clinical trials.

[11]  Alessandro Liberati,et al.  Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study , 2005, BMJ : British Medical Journal.

[12]  S. Greenland.  Quality Scores Are Useless and Potentially Misleading: Reply to “Re: A Critical Look at Some Popular Analytic Methods” , 1994 .

[13]  G. Upton.  Fisher's Exact Test , 1992 .

[14]  Rachel Churchill,et al.  Development of a quality assessment instrument for trials of treatments for depression and neurosis , 2001 .

[15]  Peter C Gøtzsche,et al.  Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review , 2006, BMJ : British Medical Journal.

[16]  G A Colditz,et al.  Estrogen replacement therapy and coronary heart disease: a quantitative assessment of the epidemiologic evidence. , 2004, Preventive medicine.

[17]  M. Egger,et al.  The hazards of scoring the quality of clinical trials for meta-analysis. , 1999, JAMA.

[18]  Douglas G Altman,et al.  Systematic reviews in health care: Assessing the quality of controlled clinical trials. , 2001, BMJ.

[19]  A. Hrõbjartsson,et al.  Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. , 2004, JAMA.

[20]  Christian Gluud,et al.  Reported Methodologic Quality and Discrepancies between Large and Small Randomized Trials in Meta-Analyses , 2001, Annals of Internal Medicine.

[21]  B. Beermann,et al.  Evidence b(i)ased medicine—selective reporting from studies sponsored by pharmaceutical industry: review of studies in new drug applications , 2003, BMJ : British Medical Journal.

[22]  Cindy Farquhar,et al.  3 The Cochrane Library , 1996 .

[23]  J. Manson,et al.  Postmenopausal estrogen therapy and cardiovascular disease. Ten-year follow-up from the nurses' health study. , 1991, The New England journal of medicine.

[24]  Andrea Furlan,et al.  Updated Method Guidelines for Systematic Reviews in the Cochrane Collaboration Back Review Group , 2003, Spine.

[25]  D. Cook,et al.  Assessing the quality of reports of randomised trials: implications for the conduct of meta-analyses. , 1999, Health technology assessment.

[26]  Charles Kooperberg,et al.  Risks and benefits of estrogen plus progestin in healthy postmenopausal women: principal results From the Women's Health Initiative randomized controlled trial. , 2002, JAMA.

[27]  H. Melander.  [Selective reporting--greater problem than selective publishing?]. , 2005, Läkartidningen.

[28]  A R Jadad,et al.  Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. , 1998, JAMA.

[29]  G. Guyatt,et al.  Grading quality of evidence and strength of recommendations , 2004, BMJ : British Medical Journal.

[30]  D. Altman.  Comparability of Randomised Groups , 1985 .

[31]  B. Charlton,et al.  The uses and abuses of meta-analysis. , 1996, Family practice.

[32]  Paul Glasziou,et al.  Assessing the quality of research , 2004, BMJ : British Medical Journal.

[33]  P. Tugwell,et al.  Evidence-Based Rheumatology , 2003 .

[34]  M. Bhandari,et al.  Quality in the reporting of randomized trials in surgery: is the Jadad scale reliable? , 2001, Controlled clinical trials.

[35]  J. Hilden,et al.  Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. , 2007, International journal of epidemiology.

[36]  D. Cook,et al.  Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? , 1998, The Lancet.

[37]  R. J. Hayes,et al.  Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. , 1995, JAMA.

[38]  P. Gøtzsche.  Methodology and overt and hidden bias in reports of 196 double-blind trials of nonsteroidal antiinflammatory drugs in rheumatoid arthritis. , 1989, Controlled clinical trials.

[39]  G. Colditz,et al.  Estrogen replacement therapy and coronary heart disease: A quantitative assessment of the epidemiologic evidence☆☆☆ , 1991 .

[40]  A R Jadad,et al.  Assessing the quality of reports of randomized clinical trials: is blinding necessary? , 1996, Controlled clinical trials.

[41]  Michele Tarsilla.  Cochrane Handbook for Systematic Reviews of Interventions , 2010, Journal of MultiDisciplinary Evaluation.

[42]  J D Emerson,et al.  An empirical study of the possible relation of treatment differences to quality scores in controlled randomized clinical trials. , 1990, Controlled clinical trials.

[43]  A. Feinstein,et al.  Clinical Epidemiology: The Architecture of Clinical Research. , 1987 .