A Residual-Based Approach to Validate Q-Matrix Specifications

Q-matrix validation is of increasing concern due to the significance and subjective tendency of Q-matrix construction in the modeling process. This research proposes a residual-based approach to empirically validate Q-matrix specification based on a combination of fit measures. The approach separates Q-matrix validation into four logical steps, including the test-level evaluation, possible distinction between attribute-level and item-level misspecifications, identification of the hit item, and fit information to aid in item adjustment. Through simulation studies and real-life examples, it is shown that the misspecified items can be detected as the hit item and adjusted sequentially when the misspecification occurs at the item level or at random. Adjustment can be based on the maximum reduction of the test-level measures. When adjustment of individual items tends to be useless, attribute-level misspecification is of concern. The approach can accommodate a variety of cognitive diagnosis models (CDMs) and be extended to cover other response formats.

[1]  Jimmy de la Torre,et al.  A General Cognitive Diagnosis Model for Expert-Defined Polytomous Attributes , 2013 .

[2]  J. Templin,et al.  The Effects of Q-Matrix Misspecification on Parameter Estimates and Classification Accuracy in the DINA Model , 2008 .

[3]  Chia-Yi Chiu,et al.  A Procedure for Assessing the Completeness of the Q-Matrices of Cognitively Diagnostic Tests , 2017, Psychometrika.

[4]  Jinsong Chen,et al.  Exploring reading comprehension skill relationships through the G-DINA model , 2016 .

[5]  K. Tatsuoka.  RULE SPACE: AN APPROACH FOR DEALING WITH MISCONCEPTIONS BASED ON ITEM RESPONSE THEORY , 1983 .

[6]  Chia-Yi Chiu.  Statistical Refinement of the Q-Matrix in Cognitive Diagnosis , 2013 .

[7]  Matthias von Davier,et al.  A general diagnostic model applied to language testing data. , 2008, The British journal of mathematical and statistical psychology.

[8]  L. T. DeCarlo.  On the Analysis of Fraction Subtraction Data: The DINA Model, Classification, Latent Class Sizes, and the Q-Matrix , 2011 .

[9]  Jingchen Liu,et al.  Data-Driven Learning of Q-Matrix , 2012, Applied psychological measurement.

[10]  Curtis Tatsuoka,et al.  Data analytic methods for latent partially ordered classification models , 2002 .

[11]  Z. Ying,et al.  Statistical Analysis of Q-Matrix Based Diagnostic Classification Models , 2015, Journal of the American Statistical Association.

[12]  Gongjun Xu,et al.  Identifiability of Diagnostic Classification Models , 2015, Psychometrika.

[13]  John T. Willse,et al.  Defining a Family of Cognitive Diagnosis Models Using Log-Linear Models with Latent Variables , 2009 .

[14]  J. D. L. Torre,et al.  The Generalized DINA Model Framework. , 2011 .

[15]  K. Tatsuoka.  Toward an Integration of Item-Response Theory and Cognitive Error Diagnosis. , 1987 .

[16]  Jimmy de la Torre,et al.  Relative and Absolute Fit Evaluation in Cognitive Diagnosis Modeling. , 2013 .

[17]  John Cresswell,et al.  Assessing scientific, reading and mathematical literacy : a framework for PISA 2006 , 2006 .

[18]  Jeffrey A Douglas,et al.  Higher-order latent trait models for cognitive diagnosis , 2004 .

[19]  J. D. L. Torre,et al.  DINA Model and Parameter Estimation: A Didactic , 2009 .

[20]  Chia-Yi Chiu,et al.  Cluster Analysis for Cognitive Diagnosis: Theory and Applications , 2009 .

[21]  L. T. DeCarlo.  Recognizing Uncertainty in the Q-Matrix via a Bayesian Extension of the DINA Model , 2012 .

[22]  Chia-Yi Chiu,et al.  A General Method of Empirical Q-matrix Validation , 2016, Psychometrika.

[23]  Jimmy de la Torre,et al.  An Empirically Based Method of Q‐Matrix Validation for the DINA Model: Development and Applications , 2008 .

[24]  E. Maris.  Estimating multiple classification latent class models , 1999 .

[25]  Andreas Schleicher.  Measuring Student Knowledge and Skills: A New Framework for Assessment. , 1999 .

[26]  J. Templin,et al.  Measurement of psychological disorders using cognitive diagnosis models. , 2006, Psychological methods.

[27]  B. Junker,et al.  Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory , 2001 .

[28]  Sarah M. Hartz,et al.  A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. , 2002 .