Meta‐research: The art of getting it wrong

Meta-analysis has major strengths, but sometimes it can often lead to wrong and misleading answers. In this SRSM presidential address, I discuss some case studies that exemplify these problems, including examples from meta-analyses of both clinical trials and observational associations. I also discuss issues of effect size estimation, bias (in particular significance-chasing biases), and credibility in meta-research. I examine the factors that affect the credibility of meta-analyses, including magnitude of effects, multiplicity of analyses, scale of data, flexibility of analyses, reporting, and conflicts of interest. Under the current circumstances, a survey of expert meta-analysts attending the SRSM meeting showed that most of them believe that the true effect is practically equally likely to lie within the 95% confidence interval of a meta-analysis or outside of it. Finally, I address the placement of meta-analysis in the wider current research agenda and make a plea for adoption of more prospective meta-designs. In many/most/all fields, all primary original research may be designed, executed, and interpreted as a prospective meta-analysis. Copyright © 2011 John Wiley & Sons, Ltd.

[1]  J. Ioannidis,et al.  Percutaneous Coronary Intervention Versus Conservative Therapy in Nonacute Coronary Artery Disease: A Meta-Analysis , 2005, Circulation.

[2]  D. Matthaiou,et al.  Adjunctive dexamethasone therapy for bacterial meningitis in adults: a meta‐analysis of randomized controlled trials , 2008, European journal of neurology.

[3]  J. Ioannidis,et al.  An exploratory test for an excess of significant findings , 2007, Clinical trials.

[4]  J. Ioannidis.  Contradicted and initially stronger effects in highly cited clinical research. , 2005, JAMA.

[5]  K. Prasad,et al.  Dexamethasone treatment for acute bacterial meningitis: how strong is the evidence for routine use? , 1995, Journal of neurology, neurosurgery, and psychiatry.

[6]  John P A Ioannidis,et al.  Common genetic variants for breast cancer: 32 largely refuted candidates and larger prospects. , 2006, Journal of the National Cancer Institute.

[7]  David Moher,et al.  Epidemiology and Reporting Characteristics of Systematic Reviews , 2007, PLoS medicine.

[8]  D. Berman,et al.  Optimal medical therapy with or without PCI for stable coronary disease. , 2007, The New England journal of medicine.

[9]  M D Hughes,et al.  Practical problems in interim analyses, with particular regard to estimation. , 1989, Controlled clinical trials.

[10]  George Davey Smith,et al.  Meta-analysis of randomised controlled trials , 1997, The Lancet.

[11]  J. Ioannidis.  Why Most Discovered True Associations Are Inflated , 2008, Epidemiology.

[12]  I Olkin,et al.  Meta-analysis: reconciling the results of independent studies. , 1995, Statistics in Medicine.

[13]  Julia Koricheva,et al.  Cumulative meta–analysis: a new tool for detection of temporal trends and publication bias in ecology , 2004, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[14]  J. Ioannidis,et al.  Industry sponsorship and selection of comparators in randomized clinical trials , 2010, European journal of clinical investigation.

[15]  J. Farrar,et al.  Adjunctive dexamethasone in bacterial meningitis: a meta-analysis of individual patient data , 2011 .

[16]  P. Erwin,et al.  Corticosteroid administration and outcome of adolescents and adults with acute bacterial meningitis: a meta-analysis. , 2009, Mayo Clinic proceedings.

[17]  John P A Ioannidis,et al.  Researching Genetic Versus Nongenetic Determinants of Disease: A Comparison and Proposed Unification , 2009, Science Translational Medicine.

[18]  Rachel Churchill,et al.  Effect sizes in cumulative meta-analyses of mental health randomized trials evolved over time. , 2004, Journal of clinical epidemiology.

[19]  Dexamethasone therapy for bacterial meningitis. Results of two double-blind, placebo-controlled trials. , 1988, The New England journal of medicine.

[20]  Peter C Gøtzsche,et al.  Industry-supported meta-analyses compared with meta-analyses with non-profit or no support : Differences in methodological quality and conclusions , 2015 .

[21]  D. Altman,et al.  Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors , 2005, BMJ : British Medical Journal.

[22]  D. Altman,et al.  Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research , 2004, Canadian Medical Association Journal.

[23]  Nicola J Cooper,et al.  Evidence‐based sample size calculations based upon updated meta‐analysis , 2007, Statistics in medicine.

[24]  A. Hrõbjartsson,et al.  Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. , 2004, JAMA.

[25]  P. Coyle,et al.  Glucocorticoids in central nervous system bacterial infection. , 1999, Archives of neurology.

[26]  M. Jennions,et al.  Relationships fade with time: a meta-analysis of temporal trends in publication in ecology and evolution , 2002, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[27]  A. Tonkin,et al.  Why do phase III trials of promising heart failure drugs often fail? The contribution of "regression to the truth". , 2003, Journal of cardiac failure.

[28]  J. Ioannidis,et al.  Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias , 2008, PloS one.

[29]  Larry V. Hedges,et al.  Selection method approaches to publication bias , 2005 .

[30]  R. Simes,et al.  Publication bias: evidence of delayed publication in a cohort study of clinical research projects , 1997, BMJ.

[31]  M. Khoury,et al.  Human genome epidemiologic reviews: the beginning of something HuGE. , 2000, American journal of epidemiology.

[32]  J. Ioannidis,et al.  Local Literature Bias in Genetic Epidemiology: An Empirical Evaluation of the Chinese Literature , 2005, PLoS medicine.

[33]  John P A Ioannidis,et al.  Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects. , 2011, Journal of clinical epidemiology.

[34]  J. Ioannidis,et al.  Challenges in meta-analysis of randomized clinical trials for rare harmful cardiovascular events: the case of rosiglitazone. , 2008, American heart journal.

[35]  Wolfgang Viechtbauer,et al.  Publication bias in meta-analysis: Prevention, assessment and adjustments , 2007, Psychometrika.

[36]  J. Pritchard,et al.  Overcoming the winner's curse: estimating penetrance parameters from case-control data. , 2007, American journal of human genetics.

[37]  J. Ioannidis.  Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. , 1998, JAMA.

[38]  Adnan Kastrati,et al.  A meta-analysis of 17 randomized trials of a percutaneous coronary intervention-based strategy in patients with stable coronary artery disease. , 2008, Journal of the American College of Cardiology.

[39]  J. Ioannidis,et al.  The need to consider the wider agenda in systematic reviews and meta-analyses: breadth, timing, and depth of the evidence , 2010, BMJ : British Medical Journal.

[40]  J. Ioannidis,et al.  Persistent reservations against contradicted percutaneous coronary intervention indications: citation content analysis. , 2009, American heart journal.

[41]  S. Pocock,et al.  Stopping rules and estimation problems in clinical trials. , 1988, Statistics in medicine.

[42]  Peter C Gøtzsche,et al.  Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review , 2006, BMJ : British Medical Journal.

[43]  John P A Ioannidis,et al.  Perfect study, poor evidence: interpretation of biases preceding study design. , 2008, Seminars in hematology.

[44]  Iztok Hozo,et al.  Treatment success in cancer: new cancer treatment successes identified in phase 3 randomized controlled trials conducted by the National Cancer Institute-sponsored cooperative oncology groups, 1955 to 2006. , 2008, Archives of internal medicine.

[45]  C. Berkey,et al.  Dexamethasone as Adjunctive Therapy in Bacterial Meningitis. A Meta-Analysis of Randomized Clinical Trials Since 1988 , 1997 .

[46]  D. Goldstein,et al.  UCHL‐1 is not a Parkinson's disease susceptibility gene , 2006, Annals of neurology.

[47]  T. Chalmers,et al.  Meta-analyses of randomized controlled trials. , 1987, The New England journal of medicine.

[48]  Siobhan M. Dolan,et al.  Assessment of cumulative evidence on genetic associations: interim guidelines. , 2008, International journal of epidemiology.

[49]  John P.A. Ioannidis,et al.  Integration of evidence from multiple meta-analyses: a primer on umbrella reviews, treatment networks and multiple treatments meta-analyses , 2009, Canadian Medical Association Journal.

[50]  J. Ioannidis,et al.  Pooling research results: benefits and limitations of meta-analysis. , 1999, The Joint Commission journal on quality improvement.

[51]  C. Berkey,et al.  Dexamethasone as adjunctive therapy in bacterial meningitis. A meta-analysis of randomized clinical trials since 1988. , 1997, JAMA.

[52]  D. Goldstein,et al.  UCHL1 is a Parkinson's disease susceptibility gene. , 2004 .

[53]  David Moher,et al.  Comparison of registered and published primary outcomes in randomized controlled trials. , 2009, JAMA.

[54]  J. Ioannidis.  Why Most Published Research Findings Are False , 2005, PLoS medicine.

[55]  Thomas A Trikalinos,et al.  The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey , 2007, Canadian Medical Association Journal.

[56]  G. Guyatt,et al.  Stopping randomized trials early for benefit and estimation of treatment effects: systematic review and meta-regression analysis. , 2010, JAMA.

[57]  J. Ioannidis,et al.  Why Current Publication Practices May Distort Science , 2008, PLoS medicine.

[58]  D. Cook,et al.  A guide to interpreting discordant systematic reviews. , 1997, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[59]  Peter C Gøtzsche,et al.  Is screening for breast cancer with mammography justifiable? , 2000, The Lancet.

[60]  Franz Porzsolt,et al.  The fading of reported effectiveness. A meta-analysis of randomised controlled trials , 2006, BMC medical research methodology.

[61]  John P A Ioannidis,et al.  Interpretation of tests of heterogeneity and bias in meta-analysis. , 2008, Journal of evaluation in clinical practice.

[62]  S. Willich,et al.  How objective are systematic reviews? Differences between reviews on complementary medicine. , 2003, Journal of the Royal Society of Medicine.

[63]  Carolyn Hutter,et al.  Association Between the Ubiquitin Carboxyl-Terminal Esterase L1 Gene (UCHL1) S18Y Variant and Parkinson's Disease: A HuGE Review and Meta-Analysis , 2009, American journal of epidemiology.

[64]  K. Plaisance,et al.  Prevention of Auditory Sequelae in Pediatric Bacterial Meningitis: A Meta‐Analysis , 1993, Pharmacotherapy.

[65]  J. Kaper,et al.  Health care financing systems for increasing utilisation of tobacco dependence treatment , 2003 .

[66]  J. Ioannidis,et al.  Relative Citation Impact of Various Study Designs in the Health Sciences , 2005, JAMA.

[67]  D A Katerndahl,et al.  Variability in meta-analytic results concerning the value of cholesterol reduction in coronary heart disease: a meta-meta-analysis. , 1999, American journal of epidemiology.

[68]  John P A Ioannidis,et al.  Availability of large-scale evidence on specific harms from systematic reviews of randomized trials. , 2004, The American journal of medicine.

[69]  K. Olsen,et al.  The beneficial effects of early dexamethasone administration in infants and children with bacterial meningitis. , 1991, The New England journal of medicine.

[70]  J. Ioannidis,et al.  An empirical evaluation of multifarious outcomes in pharmacogenetics: beta-2 adrenoceptor gene polymorphisms in asthma treatment , 2006, Pharmacogenetics and genomics.

[71]  M. McCarthy,et al.  Genome-wide association studies for complex traits: consensus, uncertainty and challenges , 2008, Nature Reviews Genetics.

[72]  J. Ioannidis,et al.  Origin and funding of the most frequently cited papers in medicine: database analysis , 2006, BMJ : British Medical Journal.

[73]  J. Ioannidis,et al.  Replication of past candidate loci for common diseases and phenotypes in 100 genome-wide association studies , 2010, European Journal of Human Genetics.

[74]  I. Olkin,et al.  The case of the misleading funnel plot , 2006, BMJ : British Medical Journal.

[75]  Gordon H Guyatt,et al.  Randomized trials stopped early for benefit: a systematic review. , 2005, JAMA.

[76]  Sanjay Kaul,et al.  The impact of revascularization on mortality in patients with nonacute coronary artery disease. , 2009, The American journal of medicine.

[77]  Thomas A Trikalinos,et al.  Early extreme contradictory estimates may appear in published research: the Proteus phenomenon in molecular genetics research and randomized trials. , 2005, Journal of clinical epidemiology.

[78]  B. J. Geiman,et al.  Dexamethasone and bacterial meningitis. A meta-analysis of randomized controlled trials. , 1992, The Western journal of medicine.

[79]  S. Nissen,et al.  Effect of rosiglitazone on the risk of myocardial infarction and death from cardiovascular causes. , 2007, The New England journal of medicine.

[80]  Alan Cantor,et al.  The uncertainty principle and industry-sponsored research , 2000, The Lancet.