Recognizing Uncertainty in the Q-Matrix via a Bayesian Extension of the DINA Model

In the typical application of a cognitive diagnosis model, the Q-matrix, which reflects the theory with respect to the skills indicated by the items, is assumed to be known. However, the Q-matrix is usually determined by expert judgment, and so there can be uncertainty about some of its elements. Here it is shown that this uncertainty can be recognized and explored via a Bayesian extension of the DINA (deterministic input noisy and) model. The approach used is to specify some elements of the Q-matrix as being random rather than as fixed; posterior distributions can then be used to obtain information about elements whose inclusion in the Q-matrix is questionable. Simulations show that this approach helps to recover the true Q-matrix when there is uncertainty about some elements. An application to the fraction-subtraction data of K. K. Tatsuoka suggests a modified Q-matrix that gives improved relative fit.

[1]  C. Mitchell Dayton,et al.  The Use of Probabilistic Models in the Assessment of Mastery , 1977 .

[2]  K. Tatsuoka.  Toward an Integration of Item-Response Theory and Cognitive Error Diagnosis. , 1987 .

[3]  Edward H. Haertel.  Using restricted latent class models to map the skill structure of achievement items , 1989 .

[4]  L. Joseph,et al.  Bayesian Statistics: An Introduction , 1989 .

[5]  Neil A. Macmillan,et al.  Detection Theory: A User's Guide , 1991 .

[6]  Charles J. Geyer,et al.  Practical Markov Chain Monte Carlo , 1992 .

[7]  J. Hagenaars.  Loglinear Models with Latent Variables , 1993 .

[8]  Robert J. Mislevy,et al.  TEST THEORY RECONCEIVED , 1994 .

[9]  Jeroen K. Vermunt,et al.  LEM: A general program for the analysis of categorical data. Users manual , 1997 .

[10]  Jeroen K. Vermunt,et al.  'EM: A general program for the analysis of categorical data 1 , 1997 .

[11]  Identifiers California,et al.  Annual Meeting of the National Council on Measurement in Education , 1998 .

[12]  E. Maris.  Estimating multiple classification latent class models , 1999 .

[13]  B. Junker,et al.  Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory , 2001 .

[14]  Curtis Tatsuoka,et al.  Data analytic methods for latent partially ordered classification models , 2002 .

[15]  Jeffrey A Douglas,et al.  Higher-order latent trait models for cognitive diagnosis , 2004 .

[16]  Neil A. Macmillan,et al.  Detection theory: A user's guide, 2nd ed. , 2005 .

[17]  Mladen A. Vouk,et al.  Experimental Analysis of the Q-Matrix Method in Knowledge Discovery , 2005, ISMIS.

[18]  Kenneth R. Koedinger,et al.  Learning Factors Analysis - A General Method for Cognitive Model Evaluation and Improvement , 2006, Intelligent Tutoring Systems.

[19]  Louis V. DiBello,et al.  31A Review of Cognitively Diagnostic Assessment and a Summary of Psychometric Models , 2006 .

[20]  J. Templin,et al.  The Effects of Q-Matrix Misspecification on Parameter Estimates and Classification Accuracy in the DINA Model , 2008 .

[21]  R. Henson,et al.  Cognitive Diagnostic Attribute-Level Discrimination Indices , 2008 .

[22]  Jimmy de la Torre,et al.  Model Evaluation and Multiple Strategies in Cognitive Diagnosis: An Analysis of Fraction Subtraction Data , 2008 .

[23]  Jonathan Templin,et al.  Robustness of Hierarchical Modeling of Skill Association in Cognitive Diagnosis Models , 2008 .

[24]  J. Templin,et al.  Unique Characteristics of Diagnostic Classification Models: A Comprehensive Review of the Current State-of-the-Art , 2008 .

[25]  Jimmy de la Torre,et al.  An Empirically Based Method of Q‐Matrix Validation for the DINA Model: Development and Applications , 2008 .

[26]  John T. Willse,et al.  Defining a Family of Cognitive Diagnosis Models Using Log-Linear Models with Latent Variables , 2009 .

[27]  J. D. L. Torre,et al.  DINA Model and Parameter Estimation: A Didactic , 2009 .

[28]  Lawrence T. DeCarlo,et al.  On the statistical and theoretical basis of signal detection theory and extensions: Unequal variance, random coefficient, and mixture models , 2010 .

[29]  Bengt Muthén,et al.  Bayesian SEM : A more flexible representation of substantive theory , 2010 .

[30]  L. T. DeCarlo.  On the Analysis of Fraction Subtraction Data: The DINA Model, Classification, Latent Class Sizes, and the Q-Matrix , 2011 .

[31]  Bengt Muthén,et al.  Bayesian structural equation modeling: a more flexible representation of substantive theory. , 2012, Psychological methods.