Minimum description length model selection of multinomial processing tree models

Multinomial processing tree (MPT) modeling has been widely and successfully applied as a statistical methodology for measuring hypothesized latent cognitive processes in selected experimental paradigms. In this article, we address the problem of selecting the best MPT model from a set of scientifically plausible MPT models, given observed data. We introduce a minimum description length (MDL) based model-selection approach that overcomes the limitations of existing methods such as the G2-based likelihood ratio test, the Akaike information criterion, and the Bayesian information criterion. To help ease the computational burden of implementing MDL, we provide a computer program in MATLAB that performs MDL-based model selection for any MPT model, with or without inequality constraints. Finally, we discuss applications of the MDL approach to well-studied MPT models with real data sets collected in two different experimental paradigms: source monitoring and pair clustering. The aforementioned MATLAB program may be downloaded from http://pbr.psychonomic-journals.org/content/supplemental.

[1]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[2]  Robert G. Rose,et al.  Bilingual Memory for Related and Unrelated Sentences. , 1975 .

[3]  Joel Saegert,et al.  Memory for language of input in polyglots. , 1975 .

[4]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[5]  William H. Batchelder,et al.  Separation of storage and retrieval factors in free recall of clusterable pairs. , 1980 .

[6]  William H. Batchelder,et al.  The statistical analysis of a model for storage and retrieval processes in human memory , 1986 .

[7]  David M. Riefer,et al.  Multinomial Modeling and the Measurement of Cognitive Processes. , 1988 .

[8]  Timothy R. C. Read,et al.  Goodness-Of-Fit Statistics for Discrete Multivariate Data , 1988 .

[9]  A. Shapiro.  Towards a unified theory of inequality constrained testing in multivariate analysis , 1988 .

[10]  J. Berger.  Statistical Decision Theory and Bayesian Analysis , 1988 .

[11]  G. Casella,et al.  Statistical Inference , 2003, Encyclopedia of Social Network Analysis and Mining.

[12]  David M. Riefer,et al.  Multinomial processing models of source monitoring. , 1990 .

[13]  William H. Batchelder,et al.  Statistical Inference for Multinomial Processing Tree Models , 1991 .

[14]  N. L. Johnson,et al.  Breakthroughs in Statistics , 1992 .

[15]  R. Schweickert.  A multinomial processing tree model for degradation and redintegration in immediate recall , 1993, Memory & cognition.

[16]  David M. Riefer,et al.  Response strategies in source monitoring , 1994 .

[17]  W. Batchelder,et al.  The statistical analysis of general processing tree models with the EM algorithm , 1994 .

[18]  Xiangen Hu,et al.  Measuring memory factors in source monitoring: Reply to Kinchla. , 1994 .

[19]  David M. Riefer,et al.  A multinomial modeling analysis of the recognition-failure paradigm , 1995, Memory & cognition.

[20]  Jorma Rissanen,et al.  Fisher information and stochastic complexity , 1996, IEEE Trans. Inf. Theory.

[21]  I. J. Myung,et al.  Applying Occam’s razor in modeling cognition: A Bayesian approach , 1997 .

[22]  W. Batchelder.  Multinomial processing tree models and psychological assessment. , 1998 .

[23]  R. Chechile,et al.  A New Method for Estimating Model Parameters for Multinomial Data. , 1998, Journal of mathematical psychology.

[24]  X Hu,et al.  GPT.EXE: A powerful tool for the visualization and analysis of general processing tree models , 1999, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[25]  D. Weakliem.  A Critique of the Bayesian Information Criterion for Model Selection , 1999 .

[26]  A. Raftery.  Bayes Factors and BIC , 1999 .

[27]  David M. Riefer,et al.  Theoretical and empirical review of multinomial process tree modeling , 1999, Psychonomic bulletin & review.

[28]  Wasserman,et al.  Bayesian Model Selection and Model Averaging. , 2000, Journal of mathematical psychology.

[29]  I. J. Myung,et al.  GUEST EDITORS' INTRODUCTION: Special Issue on Model Selection , 2000 .

[30]  I. J. Myung,et al.  The Importance of Complexity in Model Selection. , 2000, Journal of mathematical psychology.

[31]  Grünwald,et al.  Model Selection Based on Minimum Description Length. , 2000, Journal of mathematical psychology.

[32]  Jorma Rissanen,et al.  Strong optimality of the normalized ML models as universal codes and information in data , 2001, IEEE Trans. Inf. Theory.

[33]  M. Lee.  On the Complexity of Additive Clustering Models. , 2001, Journal of mathematical psychology.

[34]  David M. Riefer,et al.  Multinomial Modeling and the Measurement of Cognitive Processes , 2001 .

[35]  C. Raghavendra Rao,et al.  On model selection , 2001 .

[36]  Mark A. Pitt,et al.  Toward a method of selecting among computational models of cognition. , 2002 .

[37]  I. J. Myung,et al.  When a good fit can be bad , 2002, Trends in Cognitive Sciences.

[38]  David M. Riefer,et al.  Cognitive psychometrics: assessing storage and retrieval deficits in special populations with multinomial processing tree models. , 2002, Psychological assessment.

[39]  I. J. Myung,et al.  Toward a method of selecting among computational models of cognition. , 2002, Psychological review.

[40]  Richard A. Chechile,et al.  New multinomial models for the Chechile–Meyer task , 2004 .

[41]  E. Wagenmakers,et al.  AIC model selection using Akaike weights , 2004, Psychonomic bulletin & review.

[42]  Roger Ratcliff,et al.  Assessing model mimicry using the parametric bootstrap , 2004 .

[43]  Jay I. Myung,et al.  Assessing the distinguishability of models and the informativeness of data , 2004, Cognitive Psychology.

[44]  William H. Batchelder,et al.  Representing parametric order constraints in multi-trial applications of multinomial processing tree models , 2004 .

[45]  M. Lee,et al.  Common and distinctive features in stimulus similarity: A modified version of the contrast model , 2004, Psychonomic bulletin & review.

[46]  Sanjay Jain,et al.  Editors' Introduction , 2005, ALT.

[47]  Mark A. Pitt,et al.  Advances in Minimum Description Length: Theory and Applications , 2005 .

[48]  Karl Christoph Klauer,et al.  Hierarchical Multinomial Processing Tree Models: A Latent-Class Approach , 2006 .

[49]  M. Lee,et al.  Model selection for the rate problem: A comparison of significance testing, Bayesian, and minimum description length statistical inference , 2006 .

[50]  K. C. Klauer.  Hierarchical Multinomial Processing Tree Models: A Latent-Trait Approach , 2006 .

[51]  Peter Grünwald,et al.  Accumulative prediction error and the selection of time series models , 2006 .

[52]  Jay I. Myung,et al.  Model selection by Normalized Maximum Likelihood , 2006 .

[53]  Richard A. Chechile,et al.  A Model-Based Storage-Retrieval Analysis of Developmental Dyslexia. , 2007 .

[54]  William H. Batchelder,et al.  Using Multinomial Processing Tree Models to Measure Cognitive Deficits in Clinical Populations. , 2007 .

[55]  E. Wagenmakers.  A practical solution to the pervasive problems ofp values , 2007, Psychonomic bulletin & review.

[56]  Richard W. J. Neufeld,et al.  Advances in clinical cognitive science: Formal modeling of processes and symptoms. , 2007 .

[57]  P. Grünwald.  The Minimum Description Length Principle (Adaptive Computation and Machine Learning) , 2007 .

[58]  Jay I. Myung,et al.  Does response scaling cause the generalized context model to mimic a prototype model? , 2007, Psychonomic bulletin & review.

[59]  R. Schweickert,et al.  Tree inference with factors selectively influencing processes in a processing tree , 2008 .

[60]  M. Aitkin,et al.  Bayes factors: Prior sensitivity and model generalizability , 2008 .

[61]  B. Hilbig,et al.  Multinomial processing tree models: A review of the literature. , 2009 .

[62]  Anthony J Bishara,et al.  Multinomial process tree models of control and automaticity in weapon misidentification , 2009 .

[63]  Jeffrey N. Rouder,et al.  Bayesian t tests for accepting and rejecting the null hypothesis , 2009, Psychonomic bulletin & review.

[64]  W. Batchelder,et al.  A context-free language for binary multinomial processing tree models , 2009 .

[65]  William H. Batchelder,et al.  Cognitive psychometrics: Using multinomial processing tree models as measurement tools. , 2010 .

[66]  Hao Wu,et al.  On the Minimum Description Length Complexity of Multinomial Processing Tree Models. , 2010, Journal of mathematical psychology.

[67]  W. Batchelder,et al.  Beta-MPT: Multinomial processing tree models for addressing individual differences , 2010 .