Comparing time-accuracy curves: Beyond goodness-of-fit measures

The speed-accuracy trade-off (SAT) is a ubiquitous phenomenon in experimental psychology. One popular strategy for controlling SAT is to use the response signal paradigm. This paradigm produces time-accuracy curves (or SAT functions), which can be compared across different experimental conditions. The typical approach to analyzing time-accuracy curves involves the comparison of goodness-of-fit measures (e.g., adjusted-R2), as well as interpretation of point estimates. In this article, we examine the implications of this approach and discuss a number of alternative methods that have been successfully applied in the cognitive modeling literature. These methods include model selection criteria (the Akaike information criterion and the Bayesian information criterion) and interval estimation procedures (bootstrap and Bayesian). We demonstrate the utility of these methods with a hypothetical data set.

[1]  Jeffrey N. Rouder,et al.  A hierarchical model for estimating response time distributions , 2005, Psychonomic bulletin & review.

[2]  A V Reed,et al.  Speed-Accuracy Trade-Off in Recognition Memory , 1973, Science.

[3]  D. Nikolic,et al.  A tandem random walk model of the SAT paradigm: response times and accumulation of evidence. , 2002, The British journal of mathematical and statistical psychology.

[4]  Jun Lu,et al.  An introduction to Bayesian hierarchical models with an application in the theory of signal detection , 2005, Psychonomic bulletin & review.

[5]  Jeff Gill,et al.  Bayesian Methods : A Social and Behavioral Sciences Approach , 2002 .

[6]  William A Link,et al.  Model weights and the foundations of multimodel inference. , 2006, Ecology.

[7]  Anthony C. Davison,et al.  Bootstrap Methods and Their Application , 1998 .

[8]  B McElree,et al.  Covert attention accelerates the rate of visual information processing , 2001, Proceedings of the National Academy of Sciences of the United States of America.

[9]  Michael D. Lee,et al.  A Bayesian analysis of retention functions , 2004 .

[10]  Amy Wenzel,et al.  One hundred years of forgetting: A quantitative description of retention , 1996 .

[11]  Marisa Carrasco,et al.  Temporal performance fields: visual and attentional factors , 2004, Vision Research.

[12]  G. Schwarz Estimating the Dimension of a Model , 1978 .

[13]  J. Miller,et al.  The sampling distribution of d'. , 1996, Perception & psychophysics.

[14]  Brian McElree,et al.  Proactive Interference Slows Recognition by Eliminating Fast Assessments of Familiarity. , 2007 .

[15]  Andreas Voss,et al.  Fast-dm: A free program for efficient diffusion model analysis , 2007, Behavior research methods.

[16]  Scott D. Brown,et al.  The power law repealed: The case for an exponential law of practice , 2000, Psychonomic bulletin & review.

[17]  David M. Riefer,et al.  Multinomial processing models of source monitoring. , 1990 .

[18]  T. Verguts,et al.  Assessing the informational value of parameter estimates in cognitive models , 2004, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[19]  Richard Anderson-Sprecher,et al.  Model Comparisons and R 2 , 1994 .

[20]  I. J. Myung,et al.  Toward an explanation of the power law artifact: Insights from response surface analysis , 2000, Memory & cognition.

[21]  James L. McClelland,et al.  The time course of perceptual choice: the leaky, competing accumulator model. , 2001, Psychological review.

[22]  Roger Ratcliff,et al.  Assessing model mimicry using the parametric bootstrap , 2004 .

[23]  Barbara Anne Dosher,et al.  Empirical approaches to information processing: Speed-accuracy tradeoff functions or reaction time — a reply , 1979 .

[24]  Wayne A. Wickelgren,et al.  Speed-accuracy tradeoff and information processing dynamics , 1977 .

[25]  I. J. Myung,et al.  When a good fit can be bad , 2002, Trends in Cognitive Sciences.

[26]  I. J. Myung,et al.  The Importance of Complexity in Model Selection. , 2000, Journal of mathematical psychology.

[27]  Philip L. Smith,et al.  Attention orienting and the time course of perceptual decisions: response time distributions with masked and unmasked displays , 2004, Vision Research.

[28]  Jeff Miller The sampling distribution ofd′ , 1996 .

[29]  Felix Wichmann,et al.  The psychometric function: II. Bootstrap-based confidence intervals and sampling , 2001, Perception & psychophysics.

[30]  Robert Tibshirani,et al.  An Introduction to the Bootstrap , 1994 .

[31]  John C. Schmitt,et al.  Empirical approaches to information processing: Speed-accuracy tradeoff functions or reaction time , 1977 .

[32]  D. Huber Computer simulations of the ROUSE model: An analytic simulation technique and a comparison between the error variance—covariance and bootstrap methods for estimating parameter confidence , 2006, Behavior research methods.

[33]  W. Estes,et al.  Risks of drawing inferences about cognitive processes from model fits to individual versus average performance , 2005, Psychonomic bulletin & review.

[34]  Albert T. Corbett,et al.  Associative Interference and Retrieval Dynamics in Yes-No Recall and Recognition. , 1977 .

[35]  A. V. Reed,et al.  List length and the time course of recognition in immediate memory , 1976, Memory & cognition.

[36]  B. McElree,et al.  Attended and Non-Attended States in Working Memory: Accessing Categorized Structures , 1998 .

[37]  M. Lee,et al.  Modeling individual differences in cognition , 2005, Psychonomic bulletin & review.

[38]  Neil W. Mulligan,et al.  Speed-Accuracy Trade-Offs and the Dual Process Model of Recognition Memory , 1995 .

[39]  B McElree,et al.  Working memory and focal attention. , 2001, Journal of experimental psychology. Learning, memory, and cognition.

[40]  Andrew Thomas,et al.  WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility , 2000, Stat. Comput..

[41]  J. G. Snodgrass,et al.  Pragmatics of measuring recognition memory: applications to dementia and amnesia. , 1988, Journal of experimental psychology. General.

[42]  I. J. Myung,et al.  Applying Occam’s razor in modeling cognition: A Bayesian approach , 1997 .

[43]  Barbara Anne Dosher,et al.  The effects of delay and interference: A speed-accuracy study , 1981, Cognitive Psychology.

[44]  Bradley J Wolfgang,et al.  The attentional dynamics of masked detection. , 2004, Journal of experimental psychology. Human perception and performance.

[45]  H. Bozdogan,et al.  Akaike's Information Criterion and Recent Developments in Information Complexity. , 2000, Journal of mathematical psychology.

[46]  Francis Tuerlinckx,et al.  Diffusion model analysis with MATLAB: A DMAT primer , 2008, Behavior research methods.

[47]  R. Kass Bayes Factors in Practice , 1993 .

[48]  Frank Jäkel,et al.  Bayesian inference for psychometric functions. , 2005, Journal of vision.

[49]  I. J. Myung,et al.  Toward a method of selecting among computational models of cognition. , 2002, Psychological review.

[50]  Marisa Carrasco,et al.  Attention speeds processing across eccentricity: Feature and conjunction searches , 2006, Vision Research.

[51]  B. Dosher,et al.  Serial position and set size in short-term memory: The time course of recognition , 1989 .

[52]  J. Busemeyer,et al.  Model Comparisons and Model Selections Based on Generalization Criterion Methodology. , 2000, Journal of mathematical psychology.

[53]  B. McElree,et al.  Syntactic and Thematic Processing in Sentence Comprehension: Evidence for a Temporal Dissociation , 1995 .

[54]  Songmei Han,et al.  Parallel processing in visual search asymmetry. , 2004, Journal of experimental psychology. Human perception and performance.

[55]  Richard Chechile Likelihood and posterior identification: Implications for mathematical psychology , 1977 .

[56]  Karl J. Friston,et al.  Comparing dynamic causal models , 2004, NeuroImage.

[57]  B. Dosher,et al.  Serial Retrieval Processes in the Recovery of Order Information , 1993 .

[58]  A. Heathcote,et al.  Averaging learning curves across and within participants , 2003, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[59]  David R. Anderson,et al.  Model selection and multimodel inference : a practical information-theoretic approach , 2003 .

[60]  Philip L. Smith,et al.  A comparison of sequential sampling models for two-choice reaction time. , 2004, Psychological review.

[61]  J. Kuha AIC and BIC , 2004 .

[62]  Wasserman,et al.  Bayesian Model Selection and Model Averaging. , 2000, Journal of mathematical psychology.

[63]  Barbara Anne Dosher,et al.  Degree of learning and retrieval speed: Study time and multiple exposures. , 1984 .

[64]  D. Weakliem A Critique of the Bayesian Information Criterion for Model Selection , 1999 .

[65]  E. Wagenmakers A practical solution to the pervasive problems ofp values , 2007, Psychonomic bulletin & review.

[66]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[67]  M. Aitkin,et al.  Bayes factors: Prior sensitivity and model generalizability , 2008 .

[68]  Peter Dixon,et al.  Likelihood ratios: A simple and flexible statistic for empirical psychologists , 2004, Psychonomic bulletin & review.

[69]  R. Ratcliff,et al.  Time course of item and associative information: implications for global memory models. , 1989, Journal of experimental psychology. Learning, memory, and cognition.

[70]  E Galanter,et al.  A significance test for one parameter isosensitivity functions , 1967, Psychometrika.

[71]  Philip L. Smith,et al.  Stochastic Dynamic Models of Response Time and Accuracy: A Foundational Primer. , 2000, Journal of mathematical psychology.

[72]  W. D. Penny,et al.  CHAPTER 43 – Dynamic Causal Models and Bayesian selection , 2007 .

[73]  M. Lee,et al.  Bayesian statistical inference in psychology: comment on Trafimow (2003). , 2005, Psychological review.

[74]  E. Wagenmakers,et al.  AIC model selection using Akaike weights , 2004, Psychonomic bulletin & review.

[75]  Roger Ratcliff,et al.  A Theory of Memory Retrieval. , 1978 .

[76]  Brian McElree,et al.  Accessing short-term memory with semantic and phonological information: A time-course analysis , 1996, Memory & cognition.

[77]  I. J. Myung,et al.  Tutorial on maximum likelihood estimation , 2003 .

[78]  M. Lee Three case studies in the Bayesian analysis of cognitive models , 2008, Psychonomic bulletin & review.

[79]  B. Dosher Effect of Sentence Size and Network Distance on Retrieval Speed. , 1982 .

[80]  R. Ratcliff,et al.  Estimation and interpretation of 1/fα noise in human cognition , 2004 .

[81]  Andrew J. Sinclair,et al.  Mask-dependent attentional cuing effects in visual signal detection: The psychometric function for contrast , 2004, Perception & psychophysics.

[82]  A. Raftery Bayesian Model Selection in Social Research , 1995 .

[83]  R. Ratcliff Modeling response signal and response time data , 2006, Cognitive Psychology.

[84]  Robin D. Thomas Processing time predictions of current models of perception in the classic additive factors paradigm , 2006 .