The experiment is just as important as the likelihood in understanding the prior : A cautionary note on robust cognitive modelling

Cognitive modelling shares many features with statistical modelling, making it seem trivial to borrow from the practices of robust Bayesian statistics to protect the practice of robust cognitive modelling. We take one aspect of statistical workflow-prior predictive checks-and explore how they might be applied to a cognitive modelling task. We find that it is not only the likelihood that is needed to interpret the priors, we also need to incorporate experiment information as well. This suggests that while cognitive modelling might borrow from statistical practices, especially workflow, care must be made to make the adaptions necessary.

[1]  Eric-Jan Wagenmakers,et al.  Rejoinder: More Limitations of Bayesian Leave-One-Out Cross-Validation , 2018, Computational brain & behavior.

[2]  Alexander Etz,et al.  Robust Modeling in Cognitive Science , 2019, Computational Brain & Behavior.

[3]  M. Lee,et al.  Determining informative priors for cognitive models , 2018, Psychonomic bulletin & review.

[4]  Christopher W. Myers,et al.  Model flexibility analysis. , 2015, Psychological review.

[5]  M. Lee,et al.  Bayesian Cognitive Modeling: A Practical Course , 2014 .

[6]  E. Wagenmakers,et al.  Cognitive model decomposition of the BART: Assessment and application , 2011 .

[7]  Aki Vehtari,et al.  Visualization in Bayesian workflow , 2017, Journal of the Royal Statistical Society: Series A (Statistics in Society).

[8]  D. Navarro.  Between the Devil and the Deep Blue Sea: Tensions Between Scientific Judgement and Statistical Model Selection , 2018, Computational Brain & Behavior.

[9]  Shravan Vasishth,et al.  Toward a principled Bayesian workflow in cognitive science. , 2020, Psychological methods.

[10]  Nathan J. Evans,et al.  What factors are most important in finding the best model of a psychological process? , 2020 .

[11]  Gregory L. Stuart,et al.  Evaluation of a behavioral measure of risk taking: the Balloon Analogue Risk Task (BART). , 2002, Journal of experimental psychology. Applied.

[12]  Andrew Gelman,et al.  Know your population and know your model: Using model-based regression and poststratification to generalize findings beyond the observed sample. , 2019, Psychological methods.

[13]  Andrew Gelman,et al.  The Prior Can Often Only Be Understood in the Context of the Likelihood , 2017, Entropy.

[14]  Nigel O'Brian,et al.  Generalizability Theory I , 2003 .

[15]  Aki Vehtari,et al.  Validating Bayesian Inference Algorithms with Simulation-Based Calibration , 2018, 1804.06788.

[16]  Michael D. Lee,et al.  Bayesian Methods in Cognitive Modeling , 2016 .