Bayesian statistical approaches to evaluating cognitive models.

Cognitive models aim to explain complex human behavior in terms of hypothesized mechanisms of the mind. These mechanisms can be formalized in terms of mathematical structures containing parameters that are theoretically meaningful. For example, in the case of perceptual decision making, model parameters might correspond to theoretical constructs like response bias, evidence quality, response caution, and the like. Formal cognitive models go beyond verbal models in that cognitive mechanisms are instantiated in terms of mathematics and they go beyond statistical models in that cognitive model parameters are psychologically interpretable. We explore three key elements used to formally evaluate cognitive models: parameter estimation, model prediction, and model selection. We compare and contrast traditional approaches with Bayesian statistical approaches to performing each of these three elements. Traditional approaches rely on an array of seemingly ad hoc techniques, whereas Bayesian statistical approaches rely on a single, principled, internally consistent system. We illustrate the Bayesian statistical approach to evaluating cognitive models using a running example of the Linear Ballistic Accumulator model of decision making (Brown SD, Heathcote A. The simplest complete model of choice response time: linear ballistic accumulation. Cogn Psychol 2008, 57:153-178). WIREs Cogn Sci 2018, 9:e1458. doi: 10.1002/wcs.1458 This article is categorized under: Neuroscience > Computation Psychology > Reasoning and Decision Making Psychology > Theory and Methods.

[1]  X. Zeng,et al.  Evaluating marginal likelihood with thermodynamic integration method and comparison with several other numerical methods , 2016 .

[2]  Brandon M. Turner,et al.  Approaches to Analysis in Model-based Cognitive Neuroscience. , 2017, Journal of mathematical psychology.

[3]  Thomas V. Wiecki,et al.  Model-Based Cognitive Neuroscience Approaches to Computational Psychiatry , 2015 .

[4]  John K Kruschke,et al.  Posterior predictive checks can and should be Bayesian: comment on Gelman and Shalizi, 'Philosophy and the practice of Bayesian statistics'. , 2013, The British journal of mathematical and statistical psychology.

[5]  P. Green Reversible jump Markov chain Monte Carlo computation and Bayesian model determination , 1995 .

[6]  Roger Ratcliff,et al.  Assessing model mimicry using the parametric bootstrap , 2004 .

[7]  Andrew Heathcote,et al.  Drawing conclusions from choice response time models: A tutorial using the linear ballistic accumulator , 2011 .

[8]  Jun Lu,et al.  An introduction to Bayesian hierarchical models with an application in the theory of signal detection , 2005, Psychonomic bulletin & review.

[9]  G. Schwarz Estimating the Dimension of a Model , 1978 .

[10]  John A. Nelder,et al.  A Simplex Method for Function Minimization , 1965, Comput. J..

[11]  Ben R. Newell,et al.  A Hierarchical Bayesian Modeling Approach to Searching and Stopping in Multi-Attribute Judgment , 2014, Cogn. Sci..

[12]  E. Jaynes,et al.  Confidence Intervals vs Bayesian Intervals , 1976 .

[13]  Gregory Ashby,et al.  On the Dangers of Averaging Across Subjects When Using Multidimensional Scaling or the Similarity-Choice Model , 1994 .

[14]  James L. McClelland,et al.  The time course of perceptual choice: the leaky, competing accumulator model. , 2001, Psychological review.

[15]  Irving John Good,et al.  The Interface Between Statistics and Philosophy of Science , 1988 .

[16]  T. Ando Bayesian predictive information criterion for the evaluation of hierarchical Bayesian and empirical Bayes models , 2007 .

[17]  William R. Holmes,et al.  A practical guide to the Probability Density Approximation (PDA) with improved implementation and error characterization , 2015 .

[18]  J. Geweke,et al.  Bayesian Inference in Econometric Models Using Monte Carlo Integration , 1989 .

[19]  R. Nosofsky Attention, similarity, and the identification-categorization relationship. , 1986, Journal of experimental psychology. General.

[20]  I. J. Myung,et al.  Tutorial on maximum likelihood estimation , 2003 .

[21]  Adam N. Sanborn,et al.  Model evaluation using grouped or individual data , 2008, Psychonomic bulletin & review.

[22]  M. Lee,et al.  Hierarchical diffusion models for two-choice response times. , 2011, Psychological methods.

[23]  Kentaro Katahira,et al.  How hierarchical models improve point estimates of model parameters at the individual level , 2016 .

[24]  L. Wasserman,et al.  The Selection of Prior Distributions by Formal Rules , 1996 .

[25]  Kenneth J. Malmberg,et al.  The list-length effect does not discriminate between models of recognition memory , 2015 .

[26]  Brandon M. Turner,et al.  Approximate Bayesian computation with differential evolution , 2012 .

[27]  Sumio Watanabe,et al.  Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory , 2010, J. Mach. Learn. Res..

[28]  M. Lee,et al.  Bayesian statistical inference in psychology: comment on Trafimow (2003). , 2005, Psychological review.

[29]  Andrew Gelman,et al.  Two simple examples for understanding posterior p-values whose distributions are far from uniform , 2013 .

[30]  Roger Ratcliff,et al.  Individual Differences and Fitting Methods for the Two-Choice Diffusion Model of Decision Making. , 2015, Decision.

[31]  Nial Friel,et al.  Estimating the evidence – a review , 2011, 1111.1957.

[32]  E. Wagenmakers,et al.  Bayesian hypothesis testing for psychologists: A tutorial on the Savage–Dickey method , 2010, Cognitive Psychology.

[33]  Brandon M. Turner,et al.  Hierarchical Approximate Bayesian Computation , 2013, Psychometrika.

[34]  W. K. Hastings,et al.  Monte Carlo Sampling Methods Using Markov Chains and Their Applications , 1970 .

[35]  M. Lee How cognitive modeling can benefit from hierarchical Bayesian models. , 2011 .

[36]  M. Aitkin,et al.  Bayes factors: Prior sensitivity and model generalizability , 2008 .

[37]  T. Palmeri,et al.  Modelling individual difference in visual categorization , 2016, Visual cognition.

[38]  Robert A Jacobs,et al.  Bayesian learning theory applied to human cognition. , 2011, Wiley interdisciplinary reviews. Cognitive science.

[39]  S. Gronlund,et al.  Global matching models of recognition memory: How the models match the data , 1996, Psychonomic bulletin & review.

[40]  H. Philippe,et al.  Computing Bayes factors using thermodynamic integration. , 2006, Systematic biology.

[41]  Radford M. Neal Annealed importance sampling , 1998, Stat. Comput..

[42]  Sean M. Polyn,et al.  A context maintenance and retrieval model of organizational processes in free recall. , 2009, Psychological review.

[43]  Birte U. Forstmann,et al.  A Bayesian framework for simultaneously modeling neural and behavioral data , 2013, NeuroImage.

[44]  Junni L. Zhang Comparative investigation of three Bayesian p values , 2014, Comput. Stat. Data Anal..

[45]  John K Kruschke,et al.  Bayesian data analysis. , 2010, Wiley interdisciplinary reviews. Cognitive science.

[46]  Bradley P. Carlin,et al.  Bayesian measures of model complexity and fit , 2002 .

[47]  David Lindley,et al.  That wretched prior , 2004 .

[48]  Angela Kinnell,et al.  Bayesian Analysis of Recognition Memory: The Case of the List-Length Effect , 2007 .

[49]  Brandon M. Turner,et al.  Model-based cognitive neuroscience. , 2016, Journal of mathematical psychology.

[50]  S. Chib Marginal Likelihood from the Gibbs Output , 1995 .

[51]  Richard D. Morey,et al.  A Bayesian hierarchical model for the measurement of working memory capacity , 2011 .

[52]  Philip L. Smith,et al.  A comparison of sequential sampling models for two-choice reaction time. , 2004, Psychological review.

[53]  H. Akaike A new look at the statistical model identification , 1974 .

[54]  Tom Lodewyckx,et al.  A tutorial on Bayes factor estimation with the product space method , 2011 .

[55]  Scott D. Brown,et al.  The simplest complete model of choice response time: Linear ballistic accumulation , 2008, Cognitive Psychology.

[56]  I. J. Myung,et al.  The Importance of Complexity in Model Selection. , 2000, Journal of mathematical psychology.

[57]  M. Lee,et al.  Determining informative priors for cognitive models , 2018, Psychonomic bulletin & review.

[58]  Andrew Gelman,et al.  Handbook of Markov Chain Monte Carlo , 2011 .

[59]  Scott D. Brown,et al.  Bayes factors for the linear ballistic accumulator model of decision-making , 2017, Behavior Research Methods.

[60]  N. Metropolis,et al.  Equation of State Calculations by Fast Computing Machines , 1953, Resonance.

[61]  B. Love,et al.  The myth of computational level theory and the vacuity of rational analysis , 2011, Behavioral and Brain Sciences.

[62]  Adrian F. M. Smith,et al.  Sampling-Based Approaches to Calculating Marginal Densities , 1990 .

[63]  Richard M. Shiffrin,et al.  The art of model development and testing , 1997 .

[64]  M. Lee Determining the Dimensionality of Multidimensional Scaling Representations for Cognitive Modeling. , 2001, Journal of mathematical psychology.

[65]  Aki Vehtari,et al.  Understanding predictive information criteria for Bayesian models , 2013, Statistics and Computing.

[66]  Nick Chater,et al.  Bayesian models of cognition. , 2010, Wiley interdisciplinary reviews. Cognitive science.

[67]  Michael D. Lee,et al.  A Survey of Model Evaluation Approaches With a Tutorial on Hierarchical Bayesian Methods , 2008, Cogn. Sci..

[68]  W. Vanpaemel,et al.  Prior sensitivity in theory testing: An apologia for the Bayes factor , 2010 .

[69]  Wolf Vanpaemel,et al.  Constructing informative model priors using hierarchical methods , 2011 .

[70]  Christian P. Robert,et al.  Bayesian computation: a summary of the current state, and samples backwards and forwards , 2015, Statistics and Computing.

[71]  A. Pettitt,et al.  Marginal likelihood estimation via power posteriors , 2008 .

[72]  Rob J Hyndman,et al.  Computing and Graphing Highest Density Regions , 1996 .

[73]  Jeffrey N. Rouder,et al.  The fallacy of placing confidence in confidence intervals , 2015, Psychonomic bulletin & review.

[74]  J. Skilling Nested sampling for general Bayesian computation , 2006 .

[75]  B. Efron Why Isn't Everyone a Bayesian? , 1986 .

[76]  Hal S. Stern,et al.  On the Sensitivity of Bayes Factors to the Prior Distributions , 2002 .

[77]  M. Lee,et al.  Using priors to formalize theory: Optimal attention and the generalized context model , 2012, Psychonomic bulletin & review.

[78]  M. Lee,et al.  Modeling individual differences in cognition , 2005, Psychonomic bulletin & review.

[79]  J. P. McKeone,et al.  Investigation of the widely applicable Bayesian information criterion , 2015, Stat. Comput..

[80]  James L. McClelland,et al.  Bayesian analysis of simulation-based models , 2016 .

[81]  Michael S. Pratte,et al.  Using MCMC chain outputs to efficiently estimate Bayes factors , 2011 .

[82]  John K Kruschke,et al.  Bayesian Assessment of Null Values Via Parameter Estimation and Model Comparison , 2011, Perspectives on psychological science : a journal of the Association for Psychological Science.

[83]  Jeffrey N. Rouder,et al.  A hierarchical model for estimating response time distributions , 2005, Psychonomic bulletin & review.

[84]  Michael S. Pratte,et al.  Hierarchical single- and dual-process models of recognition memory , 2011 .

[85]  M. Lee,et al.  Avoiding the dangers of averaging across subjects when using multidimensional scaling , 2003 .

[86]  Brandon M. Turner,et al.  Journal of Mathematical Psychology a Tutorial on Approximate Bayesian Computation , 2022 .

[87]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[88]  Michael Smithson,et al.  Hierarchical models of simple mechanisms underlying confidence in decision making , 2011 .

[89]  M. Lee Three case studies in the Bayesian analysis of cognitive models , 2008, Psychonomic bulletin & review.

[90]  Kensuke Okada,et al.  A Bayesian approach to modeling group and individual differences in multidimensional scaling , 2016 .

[91]  Jiqiang Guo,et al.  Stan: A Probabilistic Programming Language. , 2017, Journal of statistical software.

[92]  E. Wagenmakers,et al.  Bayesian Estimation of Multinomial Processing Tree Models with Heterogeneity in Participants and Items , 2013, Psychometrika.

[93]  Estes Wk The problem of inference from curves based on group data. , 1956 .

[94]  M. Lee,et al.  A Bayesian hierarchical mixture approach to individual differences: Case studies in selective attention and representation in category learning ☆ , 2014 .

[95]  Andrew Thomas,et al.  WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility , 2000, Stat. Comput..

[96]  Jennifer J. Richler,et al.  Visual category learning. , 2014, Wiley interdisciplinary reviews. Cognitive science.

[97]  W. Gilks,et al.  Adaptive Rejection Metropolis Sampling Within Gibbs Sampling , 1995 .

[98]  Suyog H. Chandramouli,et al.  Bayes Factors, Relations to Minimum Description Length, and Overlapping Model Classes , 2016 .

[99]  S. Chib,et al.  Marginal Likelihood From the Metropolis–Hastings Output , 2001 .

[100]  Joachim Vandekerckhove,et al.  Extending JAGS: A tutorial on adding custom distributions to JAGS (with a diffusion model example) , 2013, Behavior Research Methods.

[101]  I. J. Myung,et al.  Applying Occam’s razor in modeling cognition: A Bayesian approach , 1997 .

[102]  Thomas V. Wiecki,et al.  HDDM: Hierarchical Bayesian estimation of the Drift-Diffusion Model in Python , 2013, Front. Neuroinform..