Correction: Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature

We have empirically assessed the distribution of published effect sizes and estimated power by extracting more than 100,000 statistical records from about 10,000 cognitive neuroscience and psychology papers published during the past 5 years. The reported median effect size was d=0.93 (inter-quartile range: 0.64-1.46) for nominally statistically significant results and d=0.24 (0.11-0.42) for non-significant results. Median power to detect small, medium and large effects was 0.12, 0.44 and 0.73, reflecting no improvement through the past half-century. Power was lowest for cognitive neuroscience journals. 14% of papers reported some statistically significant results, although the respective F statistic and degrees of freedom proved that these were non-significant; p value errors positively correlated with journal impact factors. False report probability is likely to exceed 50% for the whole literature. In light of our findings the recently reported low replication success in psychology is realistic and worse performance may be expected for cognitive neuroscience.

[1]  T. Yarkoni Big Correlations in Little Studies: Inflated fMRI Correlations Reflect Low Statistical Power—Commentary on Vul et al. (2009) , 2009, Perspectives on psychological science : a journal of the Association for Psychological Science.

[2]  F. De Filippis,et al.  A Selected Core Microbiome Drives the Early Stages of Three Popular Italian Cheese Manufactures , 2014, PloS one.

[3]  P. Pollard,et al.  On the probability of making Type I errors. , 1987 .

[4]  John P. A. Ioannidis,et al.  Research: increasing value, reducing waste 2 , 2014 .

[5]  R. Tibshirani,et al.  Increasing value and reducing waste in research design, conduct, and analysis , 2014, The Lancet.

[6]  Gerd Gigerenzer,et al.  Surrogate Science , 2015 .

[7]  R. Rosenthal,et al.  Statistical power: concepts, procedures, and applications. , 1996, Behaviour research and therapy.

[8]  Joshua Carp,et al.  The secret lives of experiments: Methods reporting in the fMRI literature , 2012, NeuroImage.

[9]  A. Møller,et al.  A survey of the statistical power of research in behavioral ecology and animal behavior , 2003 .

[10]  J. Ioannidis Why Most Published Research Findings Are False , 2005, PLoS medicine.

[11]  J. Wicherts,et al.  The (mis)reporting of statistical results in psychology journals , 2011, Behavior research methods.

[12]  H. Pashler,et al.  Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition 1 , 2009, Perspectives on psychological science : a journal of the Association for Psychological Science.

[13]  Michèle B. Nuijten,et al.  The prevalence of statistical reporting errors in psychology (1985–2013) , 2015, Behavior Research Methods.

[14]  W. K. Simmons,et al.  Circular analysis in systems neuroscience: the dangers of double dipping , 2009, Nature Neuroscience.

[15]  Frank L. Schmidt,et al.  What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. , 1992 .

[16]  J. Rossi,et al.  Statistical power of psychological research: what have we gained in 20 years? , 1990, Journal of consulting and clinical psychology.

[17]  David A Harrison,et al.  Sample Size and Power Calculations using the Noncentral t-distribution , 2004 .

[18]  Jennifer J. Richler,et al.  Effect size estimates: current use, calculations, and interpretation. , 2012, Journal of experimental psychology. General.

[19]  Jacob Cohen,et al.  The statistical power of abnormal-social psychological research: a review. , 1962, Journal of abnormal and social psychology.

[20]  Brian A. Nosek,et al.  Promoting an open research culture , 2015, Science.

[21]  J. Berger,et al.  Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence , 1987 .

[22]  Brian A. Nosek,et al.  Power failure: why small sample size undermines the reliability of neuroscience , 2013, Nature Reviews Neuroscience.

[23]  Michael C. Frank,et al.  Estimating the reproducibility of psychological science , 2015, Science.

[24]  J. Ioannidis,et al.  Evolution of Reporting P Values in the Biomedical Literature, 1990-2015. , 2016, JAMA.

[25]  John P. A. Ioannidis,et al.  How to Make More Published Research True , 2014, PLoS medicine.

[26]  John Beatty,et al.  The Empire of Chance: How Probability Changed Science and Everyday Life , 1989 .

[27]  C. Klein Reliability in Cognitive Neuroscience: A Meta-Meta Analysis , 2015 .

[28]  Michèle B. Nuijten,et al.  Statistical Reporting Errors and Collaboration on Statistical Analyses in Psychological Science , 2014, PloS one.

[29]  G. Gigerenzer,et al.  Do studies of statistical power have an effect on the power of studies , 1989 .

[30]  Brian A. Nosek,et al.  Scientific Utopia , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[31]  Deena Skolnick Weisberg,et al.  The Seductive Allure of Neuroscience Explanations , 2008, Journal of Cognitive Neuroscience.

[32]  J. Ioannidis Why Most Discovered True Associations Are Inflated , 2008, Epidemiology.