Recommendations in pre-registrations and internal review board proposals promote formal power analyses but do not increase sample size

In this preregistered study, we investigated whether the statistical power of a study is higher when researchers are asked to make a formal power analysis before collecting data. We compared the sample size descriptions from two sources: (i) a sample of pre-registrations created according to the guidelines for the Center for Open Science Preregistration Challenge (PCRs) and a sample of institutional review board (IRB) proposals from Tilburg School of Behavior and Social Sciences, which both include a recommendation to do a formal power analysis, and (ii) a sample of pre-registrations created according to the guidelines for Open Science Framework Standard Pre-Data Collection Registrations (SPRs) in which no guidance on sample size planning is given. We found that PCRs and IRBs (72%) more often included sample size decisions based on power analyses than the SPRs (45%). However, this did not result in larger planned sample sizes. The determined sample size of the PCRs and IRB proposals (Md = 90.50) was not higher than the determined sample size of the SPRs (Md = 126.00; W = 3389.5, p = 0.936). Typically, power analyses in the registrations were conducted with G*power, assuming a medium effect size, α = .05 and a power of .80. Only 20% of the power analyses contained enough information to fully reproduce the results and only 62% of these power analyses pertained to the main hypothesis test in the pre-registration. Therefore, we see ample room for improvements in the quality of the registrations and we offer several recommendations to do so.

[1]  D. Fanelli.  “Positive” Results Increase Down the Hierarchy of the Sciences , 2010, PloS one.

[2]  Jacob M. Marszalek,et al.  Sample Size in Psychological Research over the Past 30 Years , 2011, Perceptual and motor skills.

[3]  Jacob Cohen,et al.  THINGS I HAVE LEARNED (SO FAR) , 1990 .

[4]  M. Lee,et al.  Statistical Evidence in Experimental Psychology , 2011, Perspectives on psychological science : a journal of the Association for Psychological Science.

[5]  Brian A. Nosek,et al.  Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015 , 2018, Nature Human Behaviour.

[6]  Michael C. Frank,et al.  Estimating the reproducibility of psychological science , 2015, Science.

[7]  Edgar Erdfelder,et al.  G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences , 2007, Behavior research methods.

[8]  Robbie C. M. van Aert,et al.  Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking , 2016, Front. Psychol..

[9]  Jacob Cohen,et al.  The statistical power of abnormal-social psychological research: a review. , 1962, Journal of abnormal and social psychology.

[10]  S. Maxwell.  The persistence of underpowered studies in psychological research: causes, consequences, and remedies. , 2004, Psychological methods.

[11]  Jelte M. Wicherts,et al.  Researchers’ Intuitions About Power in Psychological Research , 2016, Psychological science.

[12]  A. D. de Groot,et al.  The meaning of “significance” for different types of research [translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas] , 2014 .

[13]  D. Fanelli.  How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data , 2009, PloS one.

[14]  C. Chambers.  Registered Reports: A new publishing initiative at Cortex , 2013, Cortex.

[15]  Ken Kelley,et al.  Sample-Size Planning for More Accurate Statistical Power: A Method Adjusting Sample Effect Sizes for Publication Bias and Uncertainty , 2017, Psychological science.

[16]  Michèle B. Nuijten,et al.  The Replication Paradox: Combining Studies can Decrease Accuracy of Effect Size Estimates , 2015 .

[17]  Robbie C. M. van Aert,et al.  Meta-analysis using effect size distributions of only statistically significant studies. , 2015, Psychological methods.

[18]  Han L. J. van der Maas,et al.  Science Perspectives on Psychological an Agenda for Purely Confirmatory Research on Behalf Of: Association for Psychological Science , 2022 .

[19]  Marco Perugini,et al.  Safeguard Power as a Protection Against Imprecise Power Estimates , 2014, Perspectives on psychological science : a journal of the Association for Psychological Science.

[20]  Matthew S. Fritz,et al.  PSYCHOLOGICAL SCIENCE Research Article Required Sample Size to Detect the Mediated Effect , 2022 .

[21]  Denny Borsboom,et al.  PsychDisclosure.org , 2013, Perspectives on psychological science : a journal of the Association for Psychological Science.

[22]  Roberto Cubelli,et al.  Questionable research practices among italian research psychologists , 2017, PloS one.

[23]  Brian A. Nosek,et al.  The preregistration revolution , 2018, Proceedings of the National Academy of Sciences.

[24]  Hristos Doucouliagos,et al.  What Meta-Analyses Reveal About the Replicability of Psychological Research , 2018, Psychological bulletin.

[25]  T. Sterling.  Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa , 1959 .

[26]  Brian A. Nosek,et al.  Promoting an open research culture , 2015, Science.

[27]  J. Wicherts,et al.  Outlier removal, sum scores, and the inflation of the Type I error rate in independent samples t tests: the power of alternatives and recommendations. , 2014, Psychological methods.

[28]  Christopher R. Chartier,et al.  The Psychological Science Accelerator: Advancing Psychology Through a Distributed Collaborative Network , 2018, Advances in methods and practices in psychological science.

[29]  H. Pashler,et al.  Editors’ Introduction to the Special Section on Replicability in Psychological Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[30]  Brian A. Nosek,et al.  Scientific Utopia , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[31]  J. Wicherts,et al.  The Rules of the Game Called Psychological Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[32]  J. Rossi,et al.  Statistical power of articles published in three health psychology-related journals. , 2001, Health psychology : official journal of the Division of Health Psychology, American Psychological Association.

[33]  Jacob Cohen,et al.  A power primer. , 1992, Psychological bulletin.

[34]  Felix D. Schönbrodt,et al.  A Bayesian bird's eye view of ‘Replications of important results in social psychology’ , 2017, Royal Society Open Science.

[35]  K. Fiedler,et al.  Questionable Research Practices Revisited , 2016 .

[36]  S. West,et al.  A comparison of methods to test mediation and other intervening variable effects. , 2002, Psychological methods.

[37]  Brian A. Nosek,et al.  Recommendations for Increasing Replicability in Psychology † , 2013 .

[38]  J. Ioannidis,et al.  Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature , 2017, PLoS biology.

[39]  Leif D. Nelson,et al.  False-Positive Psychology , 2011, Psychological science.

[40]  Neil Malhotra,et al.  Underreporting in Psychology Experiments , 2016 .

[41]  G. Loewenstein,et al.  Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling , 2012, Psychological science.

[42]  Brian A. Nosek,et al.  Ensuring the quality and specificity of preregistrations , 2020, PLoS biology.

[43]  Theodor D. Sterling,et al.  Publication decisions revisited: the effect of the outcome of statistical tests on the decision to p , 1995 .

[44]  Brian A. Nosek,et al.  Scientific Utopia: I. Opening Scientific Communication , 2012, ArXiv.