Sample size evolution in neuroimaging research: an evaluation of highly-cited studies (1990-2012) and of latest practices (2017-2018) in high-impact journals

We evaluated 1038 of the most cited structural and functional (fMRI) magnetic resonance brain imaging papers (1161 studies) published during 1990-2012 and 273 papers (302 studies) published in top neuroimaging journals in 2017 and 2018. 96% of highly cited experimental fMRI studies had a single group of participants and these studies had median sample size of 12, highly cited clinical fMRI studies (with patient participants) had median sample size of 14.5, and clinical structural MRI studies had median sample size of 50. The sample size of highly cited experimental fMRI studies increased at a rate of 0.74 participant/year and this rate of increase was commensurate with the median sample sizes of neuroimaging studies published in top neuroimaging journals in 2017 (23 participants) and 2018 (24 participants). Only 4 of 131 papers in 2017 and 5 of 142 papers in 2018 had pre-study power calculations, most for single t-tests and correlations. Only 14% of highly cited papers reported the number of excluded participants whereas about 45% of papers in 2017 and 2018 reported excluded participants. Targeted interventions from publishers and funders could facilitate increase in sample sizes and adherence to better standards.

[1]  J. Ioannidis.  Contradicted and initially stronger effects in highly cited clinical research. , 2005, JAMA.

[2]  C. Begley,et al.  Drug development: Raise standards for preclinical cancer research , 2012, Nature.

[3]  Tor D Wager,et al.  The relation between statistical power and inference in fMRI , 2017, PloS one.

[4]  J. Ioannidis.  Why Most Discovered True Associations Are Inflated , 2008, Epidemiology.

[5]  J. Ioannidis,et al.  Potential Reporting Bias in fMRI Studies of the Brain , 2013, PloS one.

[6]  Leif D. Nelson,et al.  False-Positive Psychology , 2011, Psychological science.

[7]  R. Tibshirani,et al.  Increasing value and reducing waste in research design, conduct, and analysis , 2014, The Lancet.

[8]  Michael Ingre,et al.  Why small low-powered studies are worse than large high-powered studies and how to protect against “trivial” findings in research: Comment on Friston (2012) , 2013, NeuroImage.

[9]  Thomas E. Nichols,et al.  Scanning the horizon: towards transparent and reproducible neuroimaging research , 2016, Nature Reviews Neuroscience.

[10]  Denes Szucs,et al.  A Tutorial on Hunting Statistical Significance by Chasing N , 2016, Front. Psychol..

[11]  Thomas E. Nichols,et al.  Statistical limitations in functional neuroimaging. II. Signal detection and statistical inference. , 1999, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[12]  Brian A. Nosek,et al.  Power failure: why small sample size undermines the reliability of neuroscience , 2013, Nature Reviews Neuroscience.

[13]  Joshua Carp,et al.  The secret lives of experiments: Methods reporting in the fMRI literature , 2012, NeuroImage.

[14]  John P. A. Ioannidis,et al.  A manifesto for reproducible science , 2017, Nature Human Behaviour.

[15]  Brian A. Nosek,et al.  Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention , 2014, Trends in Cognitive Sciences.

[16]  John P A Ioannidis,et al.  Meta‐research: The art of getting it wrong , 2010, Research synthesis methods.

[17]  S. Hayasaka,et al.  Power and sample size calculation for neuroimaging studies by non-central random field theory , 2007, NeuroImage.

[18]  L. Hedges,et al.  Introduction to Meta‐Analysis , 2009, International Coaching Psychology Review.

[19]  C. Glenn Begley,et al.  Raise standards for preclinical cancer research , 2012 .

[20]  J. Ioannidis.  Molecular evidence‐based medicine , 2007, European journal of clinical investigation.

[21]  Daniel R. Little,et al.  Small is beautiful: In defense of the small-N design , 2018, Psychonomic Bulletin & Review.

[22]  Jennifer J. Richler,et al.  Effect size estimates: current use, calculations, and interpretation. , 2012, Journal of experimental psychology. General.

[23]  Nick F. Ramsey,et al.  Within-subject variation in BOLD-fMRI signal changes across repeated measurements: Quantification and implications for sample size , 2008, NeuroImage.

[24]  J. Ioannidis,et al.  Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature , 2017, PLoS biology.

[25]  Feng Li,et al.  An Introduction to Metaanalysis , 2005 .

[26]  J. Ioannidis.  Why Most Published Research Findings Are False , 2005, PLoS medicine.

[27]  Gary H Glover,et al.  Estimating sample size in functional MRI (fMRI) neuroimaging studies: Statistical power analyses , 2002, Journal of Neuroscience Methods.

[28]  G. Gigerenzer,et al.  Do studies of statistical power have an effect on the power of studies , 1989 .

[29]  T. Yarkoni.  Big Correlations in Little Studies: Inflated fMRI Correlations Reflect Low Statistical Power—Commentary on Vul et al. (2009) , 2009, Perspectives on psychological science : a journal of the Association for Psychological Science.

[30]  E. S. Pearson,et al.  On the Problem of the Most Efficient Tests of Statistical Hypotheses , 1933 .

[31]  John P. A. Ioannidis,et al.  Research: increasing value, reducing waste 2 , 2014 .

[32]  E. S. Pearson,et al.  On the Problem of the Most Efficient Tests of Statistical Hypotheses , 1933 .

[33]  R. McGrath,et al.  When effect sizes disagree: the case of r and d. , 2006, Psychological methods.

[34]  J. Ioannidis,et al.  When Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment , 2016, bioRxiv.

[35]  John P. A. Ioannidis,et al.  Mapping the universe of registered reports , 2018, Nature Human Behaviour.

[36]  Gerd Gigerenzer,et al.  Do Studies of Statistical Power Have an Effect on the Power of Studies? , 2004 .

[37]  J. Ioannidis,et al.  Reproducibility in Science: Improving the Standard for Basic and Preclinical Research , 2015, Circulation research.