Incorporating Quality Scores in Meta-Analysis

This paper examines the impact of quality-score weights in meta-analysis. A simulation examines the roles of study characteristics such as population effect size (ES) and its variance on the bias and mean square errors (MSEs) of the estimators for several patterns of relationship between quality and ES, and for specific patterns of systematic deviations related to quality differences. The bias and MSEs of the estimators are large when ESs from low-quality studies deviate from the population ES in specific ways, and bias does not approach zero in these cases. Because meta-analysts can never know whether biases due to quality exist, and because quality weights lead to bias in almost every condition studied, we recommend against the use of quality weights.

[1]  William R. Shadish,et al.  Combining estimates of effect size. , 1994 .

[2]  G. Ramsay,et al.  TRIPLE REGIMEN OF SELECTIVE DECONTAMINATION OF THE DIGESTIVE TRACT, SYSTEMIC CEFOTAXIME, AND MICROBIOLOGICAL SURVEILLANCE FOR PREVENTION OF ACQUIRED INFECTION IN INTENSIVE CARE , 1988, The Lancet.

[3]  M. Suarez‐Almazor,et al.  The efficacy of folic acid and folinic acid in reducing methotrexate gastrointestinal toxicity in rheumatoid arthritis. A metaanalysis of randomized controlled trials. , 1998, The Journal of rheumatology.

[4]  L. Doornbos,et al.  Selective decontamination of the digestive tract with norfloxacin in the prevention of ICU-acquired infections: a prospective randomized study , 2004, Intensive Care Medicine.

[5]  Xiao-Hua Zhou,et al.  Statistical Methods for Meta‐Analysis , 2008 .

[6]  P. Potgieter,et al.  Double-blind study of selective decontamination of the digestive tract in intensive care , 1992, The Lancet.

[7]  Douglas G Altman,et al.  Systematic reviews in health care: Assessing the quality of controlled clinical trials. , 2001, BMJ.

[8]  H. Swanson,et al.  Experimental Intervention Research on Students with Learning Disabilities: A Meta-Analysis of Treatment Outcomes , 1998 .

[9]  J. Verhoef,et al.  Prevention of colonization and infection in critically ill patients: a prospective randomized study. , 1988, Critical care medicine.

[10]  V. Deroin,et al.  Selective decontamination of the digestive tract in neurosurgical intensive care unit patients: A double‐blind, randomized, placebo‐controlled study , 1993, Critical care medicine.

[11]  D Tritchler,et al.  Modelling study quality in meta-analysis. , 1999, Statistics in medicine.

[12]  B. Quinio,et al.  Selective decontamination of the digestive tract in multiple trauma patients. A prospective double-blind, randomized, placebo-controlled study. , 1996, Chest.

[13]  D. Robert,et al.  Efficacité de la décontamination digestive sur la survenue des bactériémies nosocomiales chez les patients sous ventilation mécanique et recevant des bêtamimétiques , 1991 .

[14]  S. Chevret,et al.  A controlled trial in intensive care units of selective decontamination of the digestive tract with nonabsorbable antibiotics. The French Study Group on Selective Decontamination of the Digestive Tract. , 1992, The New England journal of medicine.

[15]  F Lemaire,et al.  Intestinal decontamination for control of nosocomial multiresistant gram-negative bacilli. Study of an outbreak in an intensive care unit. , 1989, Annals of internal medicine.

[16]  B. Schneeweiss,et al.  Oropharyngeal decontamination with gentamicin for long-term ventilated patients on stress ulcer prophylaxis with sucralfate? , 1994, Wiener klinische Wochenschrift.

[17]  J. Motin,et al.  Intestinal decontamination in a polyvalent ICU , 1990, Intensive Care Medicine.

[18]  C D Naylor,et al.  Incorporating variations in the quality of individual randomized trials into meta-analysis. , 1992, Journal of clinical epidemiology.

[19]  R. Auckenthaler,et al.  Oropharyngeal Decontamination Decreases Incidence of Ventilator-Associated Pneumonia: A Randomized, Placebo-Controlled, Double-blind Clinical Trial , 1991 .

[20]  Chen-Lin C. Kulik,et al.  The Instructional Effect of Feedback in Test-Like Events , 1991 .

[21]  Penny Whiting,et al.  Bmc Medical Research Methodology Open Access No Role for Quality Scores in Systematic Reviews of Diagnostic Accuracy Studies , 2005 .

[22]  Ethan M Balk,et al.  Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. , 2002, JAMA.

[23]  Harris Cooper,et al.  A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: the Study Design and Implementation Assessment Device (Study DIAD). , 2008, Psychological methods.

[24]  A. Walker,et al.  The impact of research quality and study design on epidemiologic estimates of the effect of nonsteroidal anti-inflammatory drugs on upper gastrointestinal tract disease. , 1992, Archives of internal medicine.

[25]  Julia H. Littell,et al.  Study Quality Assessment in Systematic Reviews of Research on Intervention Effects , 2009 .

[26]  L. Moses,et al.  Nonoptimally Weighted Least Squares , 1988 .

[27]  M. García,et al.  Effectiveness and cost of selective decontamination of the digestive tract in critically ill intubated patients. A randomized, double-blind, placebo-controlled, multicenter trial. , 1998 .

[28]  G. Bravo,et al.  Meta-analysis of the effectiveness of physical activity for the prevention of bone loss in postmenopausal women , 2005, Osteoporosis International.

[29]  B. Becker,et al.  Parental factors and the young child's postdivorce adjustment: a meta-analysis with implications for parenting arrangements. , 2000, Journal of family psychology : JFP : journal of the Division of Family Psychology of the American Psychological Association.

[30]  A. Dauber,et al.  Decrease in nosocomial pneumonia in ventilated patients by selective oropharyngeal decontamination (SOD) , 1997, Intensive Care Medicine.

[31]  Karl R. White,et al.  Does Research Support Claims About the Benefits of Involving Parents in Early Intervention Programs? , 1992 .

[32]  A D Oxman,et al.  The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials , 1998, BMJ.

[33]  W B Jonas,et al.  Impact of study quality on outcome in placebo-controlled trials of homeopathy. , 1999, Journal of clinical epidemiology.

[34]  W. Fegeler,et al.  Effect of selective flora suppression on colonization, infection, and mortality in critically ill patients: A one‐year, prospective consecutive study , 1991, Critical care medicine.

[35]  B. Rowlands,et al.  Selective decontamination of the digestive tract: a stratified, randomized, prospective study in a mixed intensive care unit. , 1991, Surgery.

[36]  Russell V. Lenth,et al.  MeWAnalysis by the Confidence Profile Method: The Statistical Synthesis of Evidence , 1993 .

[37]  A. Bérard,et al.  Combining studies using effect sizes and quality scores: application to bone loss in postmenopausal women. , 1998, Journal of clinical epidemiology.

[38]  P. Mcclelland,et al.  Reducing sepsis in severe combined acute renal and respiratory failure by selective decontamination of the digestive tract. , 1990, Critical care medicine.

[39]  J. Fleiss,et al.  Meta-analysis in epidemiology, with special reference to studies of the association between exposure to environmental tobacco smoke and lung cancer: a critique. , 1991, Journal of clinical epidemiology.

[40]  L. Hedges,et al.  The Handbook of Research Synthesis , 1995 .

[41]  R. Rodríguez-Roisín,et al.  Utility of Selective Digestive Decontamination in Mechanically Ventilated Patients , 1994, Annals of Internal Medicine.

[42]  Peter Herbison,et al.  Adjustment of meta-analyses on the basis of quality scores should be abandoned. , 2006, Journal of clinical epidemiology.

[43]  A. Carrillo,et al.  Prevention of nosocomial lung infection in ventilated patients: use of an antimicrobial pharyngeal nonabsorbable paste. , 1990, Critical care medicine.

[44]  Robert E. Slavin,et al.  Perspectives on Evidence-Based Research in Education—What Works? Issues in Synthesizing Educational Program Evaluations , 2008 .

[45]  R. J. Hayes,et al.  Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. , 1995, JAMA.

[46]  B. Bermejo,et al.  Prevention of nosocomial infection in mechanically ventilated patients: selective digestive decontamination versus sucralfate , 1997 .

[47]  F. Mosteller,et al.  How study design affects outcomes in comparisons of therapy. I: Medical. , 1989, Statistics in medicine.

[48]  R. Weinstein,et al.  A randomized, double-blind, placebo-controlled trial of selective digestive decontamination in a medical-surgical intensive care unit. , 1995, Clinical infectious diseases : an official publication of the Infectious Diseases Society of America.

[49]  D. Cook,et al.  Assessing the quality of reports of randomised trials: implications for the conduct of meta-analyses. , 1999, Health technology assessment.

[50]  M. Farnell,et al.  Prevention of Infection in Critically Ill Patients by Selective Decontamination of the Digestive Tract , 1992, Annals of Internal Medicine.

[51]  L. Hedges.  Estimation of effect size from a series of independent experiments. , 1982 .

[52]  G. Van den Berghe,et al.  Randomized, controlled trial of selective digestive decontamination in 600 mechanically ventilated patients in a multidisciplinary intensive care unit. , 1997, Critical care medicine.

[53]  F. Cerra,et al.  Selective gut decontamination reduces nosocomial infections and length of stay but not mortality or organ failure in surgical intensive care unit patients. , 1992, Archives of surgery.

[54]  G. Singer.  Meta-analysis of comparative studies of depression in mothers of children with and without developmental disabilities. , 2006, American journal of mental retardation : AJMR.

[55]  K. Peter,et al.  Prevention of colonization and respiratory infections in long-term ventilated patients by local antimicrobial prophylaxis , 2004, Intensive Care Medicine.

[56]  H. Humphreys,et al.  A controlled trial of selective decontamination of the digestive tract in intensive care and its effect on nosocomial infection. , 1992, The Journal of antimicrobial chemotherapy.

[57]  S Greenland,et al.  On the bias produced by quality scores in meta-analysis, and a hierarchical view of proposed solutions. , 2001, Biostatistics.

[58]  D. Zandstra,et al.  The effect of selective decontamination of the digestive tract on colonisation and infection rate in multiple trauma patients , 2004, Intensive Care Medicine.

[59]  N. Falchikov,et al.  Student Peer Assessment in Higher Education: A Meta-Analysis Comparing Peer and Teacher Marks , 2000 .

[60]  D. Rubin.  Meta-Analysis: Literature Synthesis or Effect-Size Surface Estimation? , 1992 .

[61]  Tasha R. Stanton,et al.  Scales to Assess the Quality of Randomized Controlled Trials: A Systematic Review , 2008, Physical Therapy.

[62]  R. Auckenthaler,et al.  Oropharyngeal decontamination decreases incidence of ventilator-associated pneumonia. A randomized, placebo-controlled, double-blind clinical trial. , 1991, JAMA.

[63]  M. Egger,et al.  The hazards of scoring the quality of clinical trials for meta-analysis. , 1999, JAMA.

[64]  M. T. Durán,et al.  Prevention of nosocomial infection in critically ill patients by selective decontamination of the digestive tract , 1992, Intensive Care Medicine.

[65]  J D Emerson,et al.  An empirical study of the possible relation of treatment differences to quality scores in controlled randomized clinical trials. , 1990, Controlled clinical trials.

[66]  Johannes B Reitsma,et al.  A systematic review finds that diagnostic reviews fail to incorporate quality despite available tools. , 2005, Journal of clinical epidemiology.

[67]  P. Lejeune,et al.  Selective intestinal decontamination in multiple trauma patients: prospective, controlled trial. , 1997, The Journal of trauma.

[68]  Jonathan R Treadwell,et al.  A system for rating the stability and strength of medical evidence , 2006, BMC medical research methodology.

[69]  F. V. van Tiel,et al.  Relationship between methodological trial quality and the effects of selective digestive decontamination on pneumonia and mortality in critically ill patients. , 2001, JAMA.

[70]  P. Onghena,et al.  Estimating the mean effect size in meta-analysis: Bias, precision, and mean squared error of different weighting methods , 2003, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[71]  T. Niyonsenga,et al.  Reliability of Chalmers' scale to assess quality in meta-analyses on pharmacological treatments for osteoporosis. , 2000, Annals of epidemiology.