An introduction to good practices in cognitive modeling

Cognitive modeling can provide important insights into the underlying causes of behavior, but the validity of those insights rests on careful model development and checking. We provide guidelines on five important aspects of the practice of cognitive modeling: parameter recovery, testing selective influence of experimental manipulations on model parameters, quantifying uncertainty in parameter estimates, testing and displaying model fit, and selecting among different model parameterizations and types of models. Each aspect is illustrated with examples.

[1]  E. Wagenmakers,et al.  Cognitive model decomposition of the BART: Assessment and application , 2011 .

[2]  Jeffrey N. Rouder,et al.  Modeling Response Times for Two-Choice Decisions , 1998 .

[3]  Stacey Wood,et al.  Risky decision making in younger and older adults: the role of learning. , 2012, Psychology and aging.

[4]  William S. Cleveland,et al.  Visualizing Data , 1993 .

[5]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[6]  I. J. Myung,et al.  Applying Occam’s razor in modeling cognition: A Bayesian approach , 1997 .

[7]  Robert G Pachella,et al.  The Interpretation of Reaction Time in Information-Processing Research 1 , 1973, Human Information Processing.

[8]  Scott D. Brown,et al.  The hare and the tortoise: emphasizing speed can change the evidence used to make decisions. , 2014, Journal of experimental psychology. Learning, memory, and cognition.

[9]  Neil A. Macmillan,et al.  Detection theory: A user's guide, 2nd ed. , 2005 .

[10]  Simon Farrell,et al.  Computational Modeling in Cognition: Principles and Practice , 2010 .

[11]  Roger Ratcliff,et al.  Assessing model mimicry using the parametric bootstrap , 2004 .

[12]  E. Wagenmakers,et al.  Diffusion versus linear ballistic accumulation: different models but the same conclusions about psychological processes? , 2010, Psychonomic bulletin & review.

[13]  Eric-Jan Wagenmakers,et al.  An EZ-diffusion model for response time and accuracy , 2007, Psychonomic bulletin & review.

[14]  Andrew Heathcote,et al.  Drawing conclusions from choice response time models: A tutorial using the linear ballistic accumulator , 2011 .

[15]  A. Heathcote,et al.  Averaging learning curves across and within participants , 2003, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[16]  Neil A. Macmillan,et al.  Detection Theory: A User's Guide , 1991 .

[17]  Philip L. Smith,et al.  Psychology and neurobiology of simple decisions , 2004, Trends in Neurosciences.

[18]  A. Raftery.  Bayesian Model Selection in Social Research , 1995 .

[19]  Richard D. Morey,et al.  Confidence Intervals from Normalized Data: A correction to Cousineau (2005) , 2008 .

[20]  G. Box.  Robustness in the Strategy of Scientific Model Building. , 1979 .

[21]  Andrew Heathcote,et al.  Linear Deterministic Accumulator Models of Simple Choice , 2012, Front. Psychology.

[22]  R. Ratcliff,et al.  1/f noise in human cognition: Is it ubiquitous, and what does it mean? , 2006, Psychonomic bulletin & review.

[23]  Bradley P. Carlin,et al.  Bayesian measures of model complexity and fit , 2002 .

[24]  Michael D. Lee,et al.  A Survey of Model Evaluation Approaches With a Tutorial on Hierarchical Bayesian Methods , 2008, Cogn. Sci..

[25]  Scott D. Brown,et al.  Quantile maximum likelihood estimation of response time distributions , 2002, Psychonomic bulletin & review.

[26]  Scott D. Brown,et al.  Domain General Mechanisms of Perceptual Decision Making in Human Cortex , 2009, The Journal of Neuroscience.

[27]  Andrew Heathcote,et al.  The form of the forgetting curve and the fate of memories , 2011 .

[28]  F. J. Anscombe,et al.  Graphs in Statistical Analysis , 1973 .

[29]  Douglas Vickers,et al.  Discriminating between the frequency of occurrence of two alternative events , 1971 .

[30]  David M. Riefer,et al.  Cognitive psychometrics: assessing storage and retrieval deficits in special populations with multinomial processing tree models. , 2002, Psychological assessment.

[31]  R. Ratcliff,et al.  Bias in the Brain: A Diffusion Model Analysis of Prior Probability and Potential Payoff , 2012, The Journal of Neuroscience.

[32]  Roger Ratcliff,et al.  A Theory of Memory Retrieval. , 1978 .

[33]  R. Ratcliff,et al.  Evaluating the unequal-variance and dual-process explanations of zROC slopes with response time data and the diffusion model , 2012, Cognitive Psychology.

[34]  Robert Tibshirani,et al.  An Introduction to the Bootstrap , 1994 .

[35]  R. Ratcliff,et al.  Aging, practice, and perceptual tasks: a diffusion model analysis. , 2006, Psychology and aging.

[36]  M. Sobel,et al.  Sociological Methodology - 2001 , 2001 .

[37]  Scott D. Brown,et al.  The simplest complete model of choice response time: Linear ballistic accumulation , 2008, Cognitive Psychology.

[38]  J T Wixted,et al.  The case against a criterion-shift account of false memory. , 2000, Psychological review.

[39]  Joachim Vandekerckhove,et al.  Oxford Handbook of Computational and Mathematical Psychology , 2014 .

[40]  Konstantinos V. Katsikopoulos.  "How to model it?" [Review of the book Cognitive modeling by J. R. Busemeyer & A. Diederich, Los Angeles: Sage, 2010] , 2011 .

[41]  Gregory L. Stuart,et al.  Evaluation of a behavioral measure of risk taking: the Balloon Analogue Risk Task (BART). , 2002, Journal of experimental psychology. Applied.

[42]  R. Ratcliff,et al.  Estimating parameters of the diffusion model: Approaches to dealing with contaminant reaction times and parameter variability , 2002, Psychonomic bulletin & review.

[43]  R. Audley,et al.  SOME ALTERNATIVE STOCHASTIC MODELS OF CHOICE1 , 1965 .

[44]  Jeffrey N Rouder,et al.  A comment on Heathcote, Brown, and Mewhort’s QMLE method for response time distributions , 2004, Psychonomic bulletin & review.

[45]  T. Ando.  Bayesian predictive information criterion for the evaluation of hierarchical Bayesian and empirical Bayes models , 2007 .

[46]  Scott D. Brown,et al.  The power law repealed: The case for an exponential law of practice , 2000, Psychonomic bulletin & review.

[47]  J. Hintze,et al.  Violin plots : A box plot-density trace synergism , 1998 .

[48]  R. Ratcliff,et al.  A Diffusion Model Account of Criterion Shifts in the Lexical Decision Task. , 2008, Journal of memory and language.

[49]  Brandon M. Turner,et al.  A method for efficiently sampling from distributions with correlated dimensions. , 2013, Psychological methods.

[50]  Roger Ratcliff,et al.  A diffusion model account of the lexical decision task. , 2004, Psychological review.

[51]  David R. Anderson,et al.  Multimodel Inference , 2004 .

[52]  A. Voss,et al.  Interpreting the parameters of the diffusion model: An empirical validation , 2004, Memory & cognition.

[53]  T. Parks.  Signal-detectability theory of recognition-memory performance. , 1966, Psychological review.

[54]  Timothy J. Pleskac,et al.  Modeling behavior in a clinically diagnostic sequential risk-taking task. , 2005, Psychological review.

[55]  R. Ratcliff,et al.  The effects of aging on reaction time in a signal detection task. , 2001, Psychology and aging.

[56]  R. Ratcliff,et al.  Sleep deprivation affects multiple distinct cognitive processes , 2009, Psychonomic bulletin & review.

[57]  P. McCullagh,et al.  Generalized Linear Models , 1984 .

[58]  A. Heathcote,et al.  Reply to Speckman and Rouder: A theoretical basis for QML , 2004 .

[59]  E. Hunt,et al.  The mathematics of behavior , 2006 .

[60]  R. Ratcliff,et al.  Modeling confidence and response time in recognition memory. , 2009, Psychological review.

[61]  Willem A. Wagenaar,et al.  Misleading postevent information: Testing parameterized models of integration in memory , 1987 .

[62]  Tom Lodewyckx,et al.  A tutorial on Bayes factor estimation with the product space method , 2011 .

[63]  David L. Gilden,et al.  Fluctuations in the Time Required for Elementary Decisions , 1997 .