Attention should be given to multiplicity issues in systematic reviews.

OBJECTIVE The objective of this paper is to describe the problem of multiple comparisons in systematic reviews and to provide some guidelines on how to deal with it in practice. STUDY DESIGN AND SETTING We describe common reasons for multiplicity in systematic reviews, and present some examples. We provide guidance on how to deal with multiplicity when it is unavoidable. RESULTS We identified six common reasons for multiplicity in systematic reviews: multiple outcomes, multiple groups, multiple time points, multiple effect measures, subgroup analyses, and multiple looks at accumulating data. The existing methods to deal with multiplicity in single trials can not always be applied in systematic reviews. CONCLUSION There is no simple and completely satisfactory solution to the problem of multiple comparisons in systematic reviews. More research is required to develop multiple comparison procedures for use in systematic reviews. Authors and consumers of systematic reviews should give serious attention to multiplicity in systematic reviews when presenting, interpreting and using the results of these reports.

[1]  S. Yusuf,et al.  Cumulating evidence from randomized trials: utilizing sequential monitoring boundaries for cumulative meta-analysis. , 1997, Controlled clinical trials.

[2]  K. Dear,et al.  Iterative generalized least squares for meta-analysis of survival data at multiple times. , 1994, Biometrics.

[3]  Peter C Austin,et al.  Testing multiple statistical hypotheses resulted in spurious associations: a study of astrological signs and health. , 2006, Journal of clinical epidemiology.

[4]  P Bauer,et al.  Multiple testing in clinical trials. , 1991, Statistics in medicine.

[5]  E. Juszczak,et al.  Compression stockings to prevent deep vein thrombosis in long-haul airline passengers. , 2006, International journal of epidemiology.

[6]  Joseph C Cappelleri,et al.  Applying the law of iterated logarithm to control type I error in cumulative meta-analysis of binary outcomes , 2007, Clinical trials.

[7]  T. Peters,et al.  Subgroup analyses in randomised controlled trials: quantifying the risks of false-positives and false-negatives. , 2001, Health technology assessment.

[8]  Simultaneous Confidence Intervals , 2005 .

[9]  Douglas G. Altman,et al.  Effect Measures for Meta‒Analysis of Trials with Binary Outcomes , 2008 .

[10]  T C Chalmers,et al.  Cumulative meta-analysis of therapeutic trials for myocardial infarction. , 1992, The New England journal of medicine.

[11]  Risk of cardiovascular events and rofecoxib: Cumulative meta-analysis , 2005 .

[12]  C. Bouza,et al.  Systematic review and meta-analysis of Transurethral Needle Ablation in symptomatic Benign Prostatic Hyperplasia , 2006, BMC urology.

[13]  Gordon H Guyatt,et al.  How strong is the evidence for the use of perioperative β blockers in non-cardiac surgery? Systematic review and meta-analysis of randomised controlled trials , 2005, BMJ : British Medical Journal.

[14]  Theo Stijnen,et al.  Combining multiple outcome measures in a meta‐analysis: an application , 2003, Statistics in medicine.

[15]  Lawrence Joseph,et al.  Impact of approximating or ignoring within‐study covariances in multivariate meta‐analyses , 2008, Statistics in medicine.

[16]  M. Parmar,et al.  Extracting summary statistics to perform meta-analyses of the published literature for survival endpoints. , 1998, Statistics in medicine.

[17]  K. Thorlund,et al.  Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis. , 2008, Journal of clinical epidemiology.

[18]  P. O'Brien.  Data and Safety Monitoring , 2005 .

[19]  Richard D Riley,et al.  Beyond the Bench: Hunting Down Fugitive Literature , 2004, Environmental Health Perspectives.

[20]  Douglas G. Altman,et al.  Systematic Reviews in Health Care: Meta-Analysis in Context: Second Edition , 2008 .

[21]  A Whitehead,et al.  A prospectively planned cumulative meta-analysis applied to a series of concurrent clinical trials. , 1997, Statistics in medicine.

[22]  K McPherson,et al.  Statistics: the problem of examining accumulating data more than once. , 1974, The New England journal of medicine.

[23]  R. Seymour,et al.  The Efficacy of Preemptive Analgesia for Acute Postoperative Pain Management: A Meta-Analysis , 2005, Anesthesia and analgesia.

[24]  P C Lambert,et al.  An evaluation of bivariate random‐effects meta‐analysis for the joint synthesis of two correlated outcomes , 2007, Statistics in medicine.

[25]  K. Ottenbacher.  Quantitative evaluation of multiplicity in epidemiology and public health research. , 1998, American journal of epidemiology.

[26]  Theo Stijnen,et al.  Advanced methods in meta‐analysis: multivariate approach and meta‐regression , 2002, Statistics in medicine.

[27]  Joseph C. Cappelleri,et al.  APPLYING THE LAW OF ITERATED LOGARITHM TO CUMULATIVE META-ANALYSIS OF A CONTINUOUS ENDPOINT , 2003 .

[28]  J. Ioannidis,et al.  Recursive cumulative meta-analysis: a diagnostic for the evolution of total randomized evidence from group and individual patient data. , 1999, Journal of clinical epidemiology.

[29]  Alexander J Sutton,et al.  What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data. , 2004, Statistics in medicine.

[30]  C H Schmid,et al.  Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care. , 1995, Journal of clinical epidemiology.

[31]  G. Smith,et al.  Going Beyond the Grand Mean: Subgroup Analysis in Meta‐Analysis of Randomised Trials , 2008 .

[32]  D. Altman,et al.  Multiple significance tests: the Bonferroni method , 1995, BMJ.

[33]  A Eisinga,et al.  Compression stockings for preventing deep vein thrombosis in airline passengers. , 2006, The Cochrane database of systematic reviews.

[34]  S. Lange,et al.  Adjusting for multiple testing--when and how? , 2001, Journal of clinical epidemiology.

[35]  S. Pocock,et al.  Subgroup analysis, covariate adjustment and baseline comparisons in clinical trial reporting: current practiceand problems , 2002, Statistics in medicine.

[36]  V. Hasselblad,et al.  Meta-analysis of Multitreatment Studies , 1998, Medical decision making : an international journal of the Society for Medical Decision Making.

[37]  J. Hsu.  Multiple Comparisons: Theory and Methods , 1996 .