Using Bayes factor hypothesis testing in neuroscience to establish evidence of absence

Most neuroscientists would agree that for brain research to progress, we have to know which experimental manipulations have no effect as much as we must identify those that do have an effect. The dominant statistical approaches used in neuroscience rely on P values and can establish the latter but not the former. This makes non-significant findings difficult to interpret: do they support the null hypothesis or are they simply not informative? Here we show how Bayesian hypothesis testing can be used in neuroscience studies to establish both whether there is evidence of absence and whether there is absence of evidence. Through simple tutorial-style examples of Bayesian t-tests and ANOVA using the open-source project JASP, this article aims to empower neuroscientists to use this approach to provide compelling and rigorous evidence for the absence of an effect. Keysers et al. show why P values do not differentiate inconclusive null findings from those that provide important evidence for the absence of an effect. They provide a tutorial on how to use Bayesian hypothesis testing to overcome this issue.

[1]  Tor D Wager,et al.  The relation between statistical power and inference in fMRI , 2017, PloS one.

[2]  D G Altman,et al.  Statistics notes: Absence of evidence is not evidence of absence , 1995 .

[3]  Jeffrey N. Rouder,et al.  Bayesian t tests for accepting and rejecting the null hypothesis , 2009, Psychonomic bulletin & review.

[4]  D. W. D.Sc.,et al.  XLII. On certain fundamental principles of scientific inquiry , 1921 .

[5]  Zoltan Dienes,et al.  Using Bayes to get the most out of non-significant results , 2014, Front. Psychol..

[6]  Alexander Etz,et al.  J. B. S. Haldane's Contribution to the Bayes Factor Hypothesis Test , 2015, 1511.08180.

[7]  Jeffrey N. Rouder,et al.  Default Bayes factors for ANOVA designs , 2012 .

[8]  Jeffrey N. Rouder,et al.  Bayes factor approaches for testing interval null hypotheses. , 2011, Psychological methods.

[9]  Jeffrey N. Rouder,et al.  Computation of Bayes Factors for Common Designs , 2015 .

[10]  Dimitris Fouskakis,et al.  Prior Distributions for Objective Bayesian Analysis , 2018, Bayesian Analysis.

[11]  M. Lee,et al.  Bayesian Benefits for the Pragmatic Researcher , 2016 .

[12]  A. Gelman,et al.  The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant , 2006 .

[13]  E. Wagenmakers,et al.  Erroneous analyses of interactions in neuroscience: a problem of significance , 2011, Nature Neuroscience.

[14]  Eric-Jan Wagenmakers,et al.  Informed Bayesian t-Tests , 2017, The American Statistician.

[15]  E. Wagenmakers,et al.  Bayesian Latent-Normal Inference for the Rank Sum Test, the Signed Rank Test, and Spearman's $\rho$ , 2017, 1712.06941.

[16]  D. W. D.Sc.,et al.  XXXVII. On certain fundamental principles of scientific inquiry. (Second Paper) , 1923 .

[17]  Jeffrey N. Rouder,et al.  Bayesian inference for psychology. Part II: Example applications with JASP , 2017, Psychonomic Bulletin & Review.

[18]  M. Lee,et al.  Determining informative priors for cognitive models , 2018, Psychonomic bulletin & review.

[19]  I. J. Myung,et al.  Applying Occam’s razor in modeling cognition: A Bayesian approach , 1997 .

[20]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[21]  Jeffrey N. Rouder,et al.  The fallacy of placing confidence in confidence intervals , 2015, Psychonomic bulletin & review.

[22]  B. Efron.  Why Isn't Everyone a Bayesian? , 1986 .

[23]  Richard D. Morey,et al.  Baysefactor: Computation of Bayes Factors for Common Designs , 2018 .

[24]  M. J. Bayarri,et al.  Criteria for Bayesian model choice with application to variable selection , 2012, 1209.5240.

[25]  J. Rouder.  Optional stopping: No problem for Bayesians , 2014, Psychonomic bulletin & review.

[26]  A. Gupta,et al.  A Bayesian Approach to , 1997 .

[27]  Jonathon Love,et al.  JASP: Graphical Statistical Software for Common Statistical Designs , 2019, Journal of Statistical Software.

[28]  J. Díez.  [A problem of significance]. , 1996, Medicina clinica.

[29]  Christopher D. Chambers,et al.  Redefine statistical significance , 2017, Nature Human Behaviour.

[30]  C. Gallistel,et al.  The Importance of Proving the Null , 2022 .

[31]  Felix D. Schönbrodt,et al.  Bayes factor design analysis: Planning for compelling evidence , 2016, Psychonomic Bulletin & Review.

[32]  H. Jeffreys.  Some Tests of Significance, Treated by the Theory of Probability , 1935, Mathematical Proceedings of the Cambridge Philosophical Society.

[33]  Christian Keysers,et al.  Emotional Mirror Neurons in the Rat’s Anterior Cingulate Cortex , 2019, Current Biology.

[34]  M. Lee,et al.  Bayesian Cognitive Modeling: A Practical Course , 2014 .

[35]  Ward Edwards,et al.  Bayesian statistical inference for psychological research. , 1963 .

[36]  Jeffrey N. Rouder,et al.  The need for Bayesian hypothesis testing in psychological science , 2017 .

[37]  Felix D. Schönbrodt,et al.  A tutorial on Bayes Factor Design Analysis using an informed prior , 2019, Behavior Research Methods.

[38]  E. Wagenmakers,et al.  Bayesian estimation of explained variance in ANOVA designs , 2019, Statistica Neerlandica.

[39]  W. W. Rozeboom.  The fallacy of the null-hypothesis significance test. , 1960, Psychological bulletin.

[40]  Jeffrey N. Rouder,et al.  Model comparison in ANOVA , 2016, Psychonomic bulletin & review.

[41]  E. Wagenmakers,et al.  Bayesian rank-based hypothesis testing for the rank sum test, the signed rank test, and Spearman's ρ , 2017, Journal of applied statistics.

[42]  H. Jeffreys,et al.  Theory of probability , 1896 .

[43]  J. Ioannidis,et al.  Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature , 2017, PLoS biology.