Recognition memory models and binary-response ROCs: A comparison by minimum description length

Model comparison in recognition memory has frequently relied on receiver operating characteristics (ROC) data. We present a meta-analysis of binary-response ROC data that builds on previous such meta-analyses and extends them in several ways. Specifically, we include more data and consider a much more comprehensive set of candidate models. Moreover, we bring to bear modern developments in model selection on the current selection problem. The new methods are based on the minimum description length framework, leading to the normalized maximum likelihood (NML) index for assessing model performance, taking into account differences between the models in flexibility due to functional form. Overall, NML results for individual ROC data indicate a preference for a discrete-state model that assumes a mixture of detection and guessing states.

[1]  K. C. Klauer,et al.  Toward a complete decision model of item and source recognition: A discrete-state approach , 2010, Psychonomic bulletin & review.

[2]  L. T. DeCarlo.  Signal detection theory with finite mixture distributions: theoretical developments with applications to recognition memory. , 2002, Psychological review.

[3]  P. Grünwald.  The Minimum Description Length Principle (Adaptive Computation and Machine Learning) , 2007 .

[4]  M. Rugg,et al.  Separating the Brain Regions Involved in Recollection and Familiarity in Recognition Memory , 2005, The Journal of Neuroscience.

[5]  K C Klauer,et al.  Unraveling social categorization in the "who said what?" paradigm. , 1998, Journal of personality and social psychology.

[6]  W. Estes,et al.  Risks of drawing inferences about cognitive processes from model fits to individual versus average performance , 2005, Psychonomic bulletin & review.

[7]  Jorma Rissanen,et al.  Strong optimality of the normalized ML models as universal codes and information in data , 2001, IEEE Trans. Inf. Theory.

[8]  Mark A. Pitt,et al.  Advances in Minimum Description Length: Theory and Applications , 2005 .

[9]  B. Hilbig,et al.  Multinomial processing tree models: A review of the literature. , 2009 .

[10]  Kristopher J Preacher,et al.  Quantifying Parsimony in Structural Equation Modeling , 2006, Multivariate behavioral research.

[11]  Akira R. O’Connor,et al.  Some Memories are Odder than Others: Judgments of Episodic Oddity Violate Known Decision Rules. , 2011, Journal of memory and language.

[12]  J. Falmagne.  Elements of psychophysical theory , 1985 .

[13]  K. C. Klauer,et al.  Assessing the belief bias effect with ROCs: reply to Dube, Rotello, and Heit (2010). , 2011, Psychological review.

[14]  J. D. Balakrishnan,et al.  Decision processes in discrimination: fundamental misrepresentations of signal detection theory. , 1999, Journal of experimental psychology. Human perception and performance.

[15]  A P Yonelinas,et al.  Consciousness, control, and confidence: the 3 Cs of recognition memory. , 2001, Journal of experimental psychology. General.

[16]  I. J. Myung,et al.  When a good fit can be bad , 2002, Trends in Cognitive Sciences.

[17]  Jay I. Myung,et al.  Model selection by Normalized Maximum Likelihood , 2006 .

[18]  T. Wickens.  Elementary Signal Detection Theory , 2001 .

[19]  Jeffrey N Rouder,et al.  An assessment of fixed-capacity models of visual working memory , 2008, Proceedings of the National Academy of Sciences.

[20]  K. C. Klauer.  Hierarchical Multinomial Processing Tree Models: A Latent-Trait Approach , 2006 .

[21]  Hao Wu,et al.  On the Minimum Description Length Complexity of Multinomial Processing Tree Models. , 2010, Journal of mathematical psychology.

[22]  R. Kinchla.  Comments on Batchelder and Riefer's multinomial model for source monitoring. , 1994, Psychological review.

[23]  R Ratcliff,et al.  Continuous versus discrete information processing modeling accumulation of partial information. , 1988, Psychological review.

[24]  F. Strack,et al.  Memory for Nonoccurrences: Metacognitive and Presuppositional Strategies , 1994 .

[25]  K. Dieussaert,et al.  Proceedings of the 26th annual conference of the cognitive science society , 2004 .

[26]  E. Erdfelder,et al.  Source discrimination, item detection, and multinomial models of source monitoring. , 1996 .

[27]  Vijay Balasubramanian,et al.  Statistical Inference, Occam's Razor, and Statistical Mechanics on the Space of Probability Distributions , 1996, Neural Computation.

[28]  C. Rotello,et al.  Binary ROCs in perception and recognition memory are curved. , 2012, Journal of experimental psychology. Learning, memory, and cognition.

[29]  Henrik Singmann,et al.  On the measurement of criterion noise in signal detection theory: the case of recognition memory. , 2012, Psychological review.

[30]  H R BLACKWELL,et al.  Neural theories of simple visual discriminations. , 1963, Journal of the Optical Society of America.

[31]  I. J. Myung,et al.  GUEST EDITORS' INTRODUCTION: Special Issue on Model Selection , 2000 .

[32]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[33]  David Kellen,et al.  Evaluating models of recognition memory using first- and second-choice responses , 2011 .

[34]  Jay I. Myung,et al.  Assessing the distinguishability of models and the informativeness of data , 2004, Cognitive Psychology.

[35]  J. Wixted.  Dual-process theory and signal-detection theory of recognition memory. , 2007, Psychological review.

[36]  T. Parks.  Signal-detectability theory of recognition-memory performance. , 1966, Psychological review.

[37]  D. M. Green,et al.  Signal detection theory and psychophysics , 1966 .

[38]  A. Bröder,et al.  Recognition ROCs are curvilinear-or are they? On premature arguments against the two-high-threshold model of recognition. , 2009, Journal of experimental psychology. Learning, memory, and cognition.

[39]  Bennet B. Murdock,et al.  Memory and the theory of signal detection. , 1970 .

[40]  J. G. Snodgrass,et al.  Pragmatics of measuring recognition memory: applications to dementia and amnesia. , 1988, Journal of experimental psychology. General.

[41]  S. Walker,et al.  On the normalized maximum likelihood and Bayesian decision theory , 2006 .

[42]  J. Rieskamp.  The probabilistic nature of preferential choice. , 2008, Journal of experimental psychology. Learning, memory, and cognition.

[43]  L. Rips,et al.  The Psychology of Survey Response , 2000 .

[44]  Jay I. Myung,et al.  Minimum description length model selection of multinomial processing tree models , 2010, Psychonomic bulletin & review.

[45]  Jean-Claude Falmagne,et al.  Mathematical psychology : current developments , 1991 .

[46]  R Ratcliff,et al.  Testing global memory models using ROC curves. , 1992, Psychological review.

[47]  K. C. Klauer,et al.  The flexibility of models of recognition memory: An analysis by the minimum-description length principle , 2011 .

[48]  Clintin P. Davis-Stober,et al.  A shift in strategy or “error”? Strategy classification over multiple stochastic specifications , 2011, Judgment and Decision Making.

[49]  J. Wixted,et al.  Testing signal-detection models of yes/no and two-alternative forced-choice recognition memory. , 2009, Journal of experimental psychology. General.

[50]  Jeffrey B. Henriques,et al.  Reward fails to alter response bias in depression. , 1994, Journal of abnormal psychology.

[51]  Jay I. Myung,et al.  Optimal experimental design for model discrimination. , 2009, Psychological review.

[52]  Petri Myllymäki,et al.  A linear-time algorithm for computing the multinomial stochastic complexity , 2007, Inf. Process. Lett..

[53]  R. Atkinson,et al.  A variable sensitivity theory of signal detection. , 1963, Psychological review.

[54]  H Pashler,et al.  How persuasive is a good fit? A comment on theory testing. , 2000, Psychological review.

[55]  Jeffrey N. Rouder,et al.  Evidence for discrete-state processing in recognition memory , 2012, Proceedings of the National Academy of Sciences.

[56]  Yong Su.  3 Minimum Description Length and Cognitive Modeling , 2003 .

[57]  Marc W Howard,et al.  Some-or-none recollection: Evidence from item and source memory. , 2010, Journal of experimental psychology. General.

[58]  M. Lee,et al.  Model selection for the rate problem: A comparison of significance testing, Bayesian, and minimum description length statistical inference , 2006 .

[59]  Neil A. Macmillan,et al.  Detection theory: A user's guide, 2nd ed. , 2005 .

[60]  Arndt Bröder,et al.  Memory for multidimensional source information. , 2002 .

[61]  Edgar Erdfelder,et al.  G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences , 2007, Behavior research methods.

[62]  David M. Riefer,et al.  Multinomial processing models of source monitoring. , 1990 .

[63]  Han L. J. van der Maas,et al.  A Phase Transition Model for the Speed-Accuracy Trade-Off in Response Time Experiments , 2011, Cogn. Sci..

[64]  H. Bozdogan,et al.  Akaike's Information Criterion and Recent Developments in Information Complexity. , 2000, Journal of mathematical psychology.

[65]  D. Navarro,et al.  NML, Bayes and true distributions: A comment on Karabatsos and Walker (2006) , 2009 .

[66]  Richard A. Chechile,et al.  New multinomial models for the Chechile–Meyer task , 2004 .

[67]  Jay I. Myung,et al.  Model Comparison Methods , 2004, Numerical Computer Methods, Part D.

[68]  M. Aitkin,et al.  Bayes factors: Prior sensitivity and model generalizability , 2008 .

[69]  A. Bröder,et al.  Validating a two-high-threshold measurement model for confidence rating data in recognition , 2013, Memory.

[70]  Roger Ratcliff,et al.  Assessing model mimicry using the parametric bootstrap , 2004 .

[71]  William H. Batchelder,et al.  Statistical Inference for Multinomial Processing Tree Models , 1991 .

[72]  M. Lee.  2 Minimum Description Length and Psychological Clustering Models , 2005 .

[73]  Lawrence T. DeCarlo,et al.  On the statistical and theoretical basis of signal detection theory and extensions: Unequal variance, random coefficient, and mixture models , 2010 .

[74]  Colleen M. Parks,et al.  Receiver operating characteristics (ROCs) in recognition memory: a review. , 2007, Psychological bulletin.

[75]  D. Hamilton,et al.  Personality attributes associated with extreme response style. , 1968, Psychological bulletin.

[76]  Colleen M. Parks,et al.  Evidence for a memory threshold in second-choice recognition memory responses , 2009, Proceedings of the National Academy of Sciences.

[77]  Jorma Rissanen,et al.  Fisher information and stochastic complexity , 1996, IEEE Trans. Inf. Theory.

[78]  I. J. Myung,et al.  Toward a method of selecting among computational models of cognition. , 2002, Psychological review.

[79]  A. Yonelinas.  Recognition memory ROCs for item and associative information: The contribution of recollection and familiarity , 1997, Memory & cognition.

[80]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[81]  E. Heit,et al.  Associative Recognition: a Case of Recall-to-reject Processing , 2022 .

[82]  R Ratcliff,et al.  A multinomial model for short-term priming in word identification. , 2001, Psychological review.

[83]  Arndt Bröder,et al.  Correction to Bröder and Schütz (2009). , 2011 .

[84]  M. Schervish.  Theory of Statistics , 1995 .

[85]  I. J. Myung,et al.  Counting probability distributions: Differential geometry and model selection , 2000, Proc. Natl. Acad. Sci. USA.

[86]  E. Heit,et al.  The belief bias effect is aptly named: a reply to Klauer and Kellen (2011). , 2011, Psychological review.

[87]  Rama Chellappa,et al.  Correction to 'Estimation and choice of Neighbors in Spatial-Interaction Models of Images' , 1983, IEEE Trans. Inf. Theory.

[88]  Verena D. Schmittmann,et al.  Multiple learning modes in the development of performance on a rule-based category-learning task , 2006, Neuropsychologia.

[89]  H. Akaike.  INFORMATION THEORY AS AN EXTENSION OF THE MAXIMUM LIKELIHOOD , 1973 .

[90]  Tim Curran,et al.  Conflict and criterion setting in recognition memory. , 2007, Journal of experimental psychology. Learning, memory, and cognition.

[91]  Jeffrey N Rouder,et al.  Latent mnemonic strengths are latent: A comment on Mickes, Wixted, and Wais (2007) , 2010, Psychonomic bulletin & review.

[92]  Daniel J. Navarro,et al.  A Note on the Applied Use of MDL Approximations , 2004, Neural Computation.

[93]  Gert Haubensak,et al.  The consistency model: A process model for absolute judgments. , 1992 .

[94]  T. Zandt.  ROC curves and confidence judgments in recognition memory. , 2000 .

[95]  David M. Riefer,et al.  Cognitive psychometrics: assessing storage and retrieval deficits in special populations with multinomial processing tree models. , 2002, Psychological assessment.

[96]  Xiangen Hu,et al.  Measuring memory factors in source monitoring: Reply to Kinchla. , 1994 .

[97]  Jun Lu,et al.  An introduction to Bayesian hierarchical models with an application in the theory of signal detection , 2005, Psychonomic bulletin & review.

[98]  R Ratcliff,et al.  Empirical generality of data from recognition memory receiver-operating characteristic functions and implications for the global memory models. , 1994, Journal of experimental psychology. Learning, memory, and cognition.

[99]  I. Dobbins,et al.  The striking similarities between standard, distractor-free, and target-free recognition , 2011, Memory & cognition.

[100]  Andrew P Yonelinas,et al.  The process-dissociation approach two decades later: Convergence, boundary conditions, and new directions , 2012, Memory & cognition.

[101]  Michael E Hasselmo,et al.  Scopolamine impairs human recognition memory: data and modeling. , 2003, Behavioral neuroscience.

[102]  J. Wixted,et al.  A continuous dual-process model of remember/know judgments. , 2010, Psychological review.

[103]  G. Mandler.  Recognizing: The judgment of previous occurrence. , 1980 .

[104]  David M. Riefer,et al.  Multinomial Modeling and the Measurement of Cognitive Processes , 2001 .

[105]  J. Wixted,et al.  The diagnosticity of individual data for model selection: Comparing signal-detection models of recognition memory , 2011, Psychonomic bulletin & review.

[106]  Lawrence T DeCarlo,et al.  The mirror effect and mixture signal detection theory. , 2007, Journal of experimental psychology. Learning, memory, and cognition.

[107]  Christian P. Robert,et al.  Monte Carlo Statistical Methods , 2005, Springer Texts in Statistics.

[108]  I. J. Myung,et al.  The Importance of Complexity in Model Selection. , 2000, Journal of mathematical psychology.

[109]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[110]  Kenneth J. Malmberg,et al.  Recognition memory: A review of the critical findings and an integrated theory for relating them , 2008, Cognitive Psychology.

[111]  S. Morgenthaler.  Robustness in Statistics , 2001 .

[112]  Riitta Jääskeläinen,et al.  Who Said What? , 2003 .

[113]  Adam N. Sanborn,et al.  Model evaluation using grouped or individual data , 2008, Psychonomic bulletin & review.

[114]  Jeffrey Annis,et al.  On the relationship between memory and perception: sequential dependencies in recognition memory testing. , 2012, Journal of experimental psychology. General.

[115]  Colleen M. Parks,et al.  Moving beyond pure signal-detection models: comment on Wixted (2007). , 2007, Psychological review.

[116]  L. M. M.-T..  Theory of Probability , 1929, Nature.

[117]  Roger Ratcliff,et al.  Beyond ROC curvature: Strength effects and response time data support continuous-evidence models of recognition memory. , 2012, Journal of memory and language.

[118]  R. Chechile,et al.  A New Method for Estimating Model Parameters for Multinomial Data. , 1998, Journal of mathematical psychology.

[119]  Jay I. Myung,et al.  Does response scaling cause the generalized context model to mimic a prototype model? , 2007, Psychonomic bulletin & review.

[120]  David R. Anderson,et al.  Model selection and multimodel inference : a practical information-theoretic approach , 2003 .

[121]  William P. Banks,et al.  Signal detection theory and human memory. , 1970 .

[122]  Jorma Rissanen,et al.  Universal coding, information, prediction, and estimation , 1984, IEEE Trans. Inf. Theory.

[123]  R. Ratcliff,et al.  Evaluating the unequal-variance and dual-process explanations of zROC slopes with response time data and the diffusion model , 2012, Cognitive Psychology.

[124]  Sanjay Jain,et al.  Editors' Introduction , 2005, ALT.

[125]  Kenneth J. Malmberg,et al.  On the form of ROCs constructed from confidence ratings. , 2002, Journal of experimental psychology. Learning, memory, and cognition.

[126]  Lawrence T DeCarlo,et al.  Process dissociation and mixture signal detection theory. , 2008, Journal of experimental psychology. Learning, memory, and cognition.

[127]  David M. Riefer,et al.  Theoretical and empirical review of multinomial process tree modeling , 1999, Psychonomic bulletin & review.