Flaws in the application and interpretation of statistical analyses in systematic reviews of therapeutic interventions were common: a cross-sectional analysis.

OBJECTIVES The objective of the study was to investigate the application and interpretation of statistical analyses in a cross-section of systematic reviews (SRs) of therapeutic interventions, without restriction by journal, clinical condition, or specialty. STUDY DESIGN AND SETTING We evaluated a random sample of SRs assembled previously, which were indexed in MEDLINE® during February 2014, focused on a treatment or prevention question, and reported at least one meta-analysis. The reported statistical methods used in each SR were extracted from articles and online appendices by one author, with a 20% random sample extracted in duplicate. RESULTS We evaluated 110 SRs; 78/110 (71%) were non-Cochrane SRs and 55/110 (50%) investigated a pharmacological intervention. The SRs presented a median of 13 (interquartile range: 5-27) meta-analytic effects. When considering the index (primary or first reported) meta-analysis of each SR, just over half (62/110 [56%]) used the random-effects model, but few (5/62 [8%]) interpreted the meta-analytic effect correctly (as the average of the intervention effects across all studies). A statistical test for funnel plot asymmetry was reported in 17/110 (15%) SRs; however, in only 4/17 (24%) did the test include the recommended number of at least 10 studies of varying size. Subgroup analyses accompanied 42/110 (38%) index meta-analyses, but findings were not interpreted with respect to a test for interaction in 29/42 (69%) cases, and the issue of potential confounding in the subgroup analyses was not raised in any SR. CONCLUSIONS There is scope for improvement in the application and interpretation of statistical analyses in SRs of therapeutic interventions. The involvement of statisticians on the SR team and establishment of partnerships between researchers with specialist expertise in SR methods and journal editors may help overcome these shortcomings.

[1]  Andrew B Forbes,et al.  Introduction to systematic reviews and meta‐analysis , 2016, Respirology.

[2]  R. Tibshirani,et al.  Increasing value and reducing waste in research design, conduct, and analysis , 2014, The Lancet.

[3]  S. Khan,et al.  Statistical controversies in clinical research: publication bias evaluations are not routinely conducted in clinical oncology systematic reviews , 2016, Annals of oncology : official journal of the European Society for Medical Oncology.

[4]  John P A Ioannidis,et al.  The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses. , 2016, The Milbank quarterly.

[5]  N. Laird,et al.  Meta-analysis in clinical trials. , 1986, Controlled clinical trials.

[6]  J. Carpenter,et al.  Meta-analytical methods to identify who benefits most from treatments: daft, deluded, or deft approach? , 2017, British Medical Journal.

[7]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[8]  S. Donegan,et al.  Cluster Randomised Trials in Cochrane Reviews: Evaluation of Methodological and Reporting Practice , 2016, PloS one.

[9]  J. Neilson,et al.  Statistical methods can be improved within Cochrane pregnancy and childbirth reviews. , 2011, Journal of clinical epidemiology.

[10]  David Moher,et al.  Mass Production of Systematic Reviews and Meta-analyses: An Exercise in Mega-silliness? , 2016, The Milbank quarterly.

[11]  Blake A. Umberham,et al.  Publication Bias and Nonreporting Found in Majority of Systematic Reviews and Meta-analyses in Anesthesiology Journals , 2016, Anesthesia and analgesia.

[12]  M. Vassar,et al.  Publication bias in dermatology systematic reviews and meta-analyses. , 2016, Journal of dermatological science.

[13]  P. Shekelle,et al.  Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement , 2015, Systematic Reviews.

[14]  D. Moher,et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement , 2009, BMJ.

[15]  Alex J Sutton,et al.  Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. , 2008, Journal of clinical epidemiology.

[16]  David Moher,et al.  Epidemiology and Reporting Characteristics of Systematic Reviews of Biomedical Research: A Cross-Sectional Study , 2016, PLoS medicine.

[17]  J. Ioannidis,et al.  Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials , 2011, BMJ : British Medical Journal.

[18]  Laura A. Levit,et al.  Finding what works in health care : standards for systematic reviews , 2011 .

[19]  C. Tudur-Smith,et al.  Exploring Treatment by Covariate Interactions Using Subgroup Analysis and Meta-Regression in Cochrane Reviews: A Review of Recent Practice , 2015, PloS one.

[20]  Douglas G Altman,et al.  Interaction revisited: the difference between two estimates , 2003, BMJ : British Medical Journal.

[21]  Douglas G. Altman,et al.  Analysing data and undertaking meta‐analyses , 2019, Cochrane Handbook for Systematic Reviews of Interventions.

[22]  James Thomas,et al.  Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews , 2016, Systematic Reviews.

[23]  Richard D Riley,et al.  Interpretation of random effects meta-analyses , 2011, BMJ : British Medical Journal.

[24]  John P A Ioannidis,et al.  Reasons or excuses for avoiding meta-analysis in forest plots , 2008, BMJ : British Medical Journal.

[25]  D. Altman,et al.  Measuring inconsistency in meta-analyses , 2003, BMJ : British Medical Journal.

[26]  G. Guyatt,et al.  How to read a systematic review and meta-analysis and apply the results to patient care: users' guides to the medical literature. , 2014, JAMA.

[27]  Jelle J Goeman,et al.  Plea for routinely presenting prediction intervals in meta-analysis , 2016, BMJ Open.

[28]  David Moher,et al.  Epidemiology and Reporting Characteristics of Systematic Reviews , 2007, PLoS medicine.

[29]  L. Hedges,et al.  Introduction to Meta‐Analysis , 2009, International Coaching Psychology Review.

[30]  I. Hambleton,et al.  The Use and Reporting of the Cross-Over Study Design in Clinical Trials and Systematic Reviews: A Systematic Assessment , 2016, PloS one.

[31]  Wasifa Zarin,et al.  Bibliographic study showed improving statistical methodology of network meta-analyses published between 1999 and 2015. , 2017, Journal of clinical epidemiology.

[32]  David Moher,et al.  Increasing value and reducing waste in biomedical research: who's listening? , 2016, The Lancet.

[33]  N. Pandis,et al.  Assessment of publication bias required improvement in oral health systematic reviews. , 2016, Journal of clinical epidemiology.

[34]  David Moher,et al.  Forest plots in reports of systematic reviews: a cross-sectional study reviewing current practice. , 2010, International journal of epidemiology.

[35]  D. Moher,et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. , 2010, International journal of surgery.

[36]  Katja Jasinskaja,et al.  Elaboration and Explanation ⋆ , 2011 .

[37]  Kurex Sidik,et al.  Simple heterogeneity variance estimation for meta‐analysis , 2005 .

[38]  D. Moher,et al.  Reporting guidance considerations from a statistical perspective: overview of tools to enhance the rigour of reporting of randomised trials and systematic reviews , 2017, Evidence-Based Mental Health.