Reconsidering Cutoff Points in the General Method of Empirical Q-Matrix Validation

Cognitive diagnosis models (CDMs) are latent class multidimensional statistical models that help classify people accurately by using a set of discrete latent variables, commonly referred to as attributes. These models require a Q-matrix that indicates the attributes involved in each item. A potential problem is that the Q-matrix construction process, typically performed by domain experts, is subjective in nature. This might lead to the existence of Q-matrix misspecifications that can lead to inaccurate classifications. For this reason, several empirical Q-matrix validation methods have been developed in the recent years. de la Torre and Chiu proposed one of the most popular methods, based on a discrimination index. However, some questions related to the usefulness of the method with empirical data remained open due the restricted number of conditions examined, and the use of a unique cutoff point (EPS) regardless of the data conditions. This article includes two simulation studies to test this validation method under a wider range of conditions, with the purpose of providing it with a higher generalization, and to empirically determine the most suitable EPS considering the data conditions. Results show a good overall performance of the method, the relevance of the different studied factors, and that using a single indiscriminate EPS is not acceptable. Specific guidelines for selecting an appropriate EPS are provided in the discussion.

[1]  Jimmy de la Torre,et al.  Analysis of Clinical Data From Cognitive Diagnosis Modeling Framework , 2015 .

[2]  M. Miller,et al.  The Impact of Q-matrix Misspecification and Model Misuse on Classification Accuracy in the Generalized DINA Model , 2017 .

[3]  J. Templin,et al.  Measurement of psychological disorders using cognitive diagnosis models. , 2006, Psychological methods.

[4]  J. Revuelta,et al.  Detection of Q-matrix misspecification using two criteria for validation of cognitive structures under the Least Squares Distance Model , 2014 .

[5]  Jimmy de la Torre,et al.  Two-Step Likelihood Ratio Test for Item-Level Model Comparison in Cognitive Diagnosis Models , 2017 .

[6]  Jingchen Liu,et al.  Data-Driven Learning of Q-Matrix , 2012, Applied psychological measurement.

[7]  Grady Hanrahan,et al.  Development and Applications of Satellite-Based Services , 2000 .

[8]  Matthias von Davier,et al.  A general diagnostic model applied to language testing data. , 2008, The British journal of mathematical and statistical psychology.

[9]  Wenchao Ma,et al.  Model Similarity, Model Selection, and Attribute Classification , 2016, Applied psychological measurement.

[10]  Chia-Yi Chiu.  Statistical Refinement of the Q-Matrix in Cognitive Diagnosis , 2013 .

[11]  Wenchao Ma,et al.  A sequential cognitive diagnosis model for polytomous responses. , 2016, The British journal of mathematical and statistical psychology.

[12]  Tiffany Barnes.  Novel Derivation and Application of Skill Matrices: The q-Matrix Method , 2010 .

[13]  E. Maris.  Estimating multiple classification latent class models , 1999 .

[14]  Christopher F. Baum.  Stata Tip 63: Modeling Proportions , 2008 .

[15]  Jimmy de la Torre,et al.  An Empirically Based Method of Q‐Matrix Validation for the DINA Model: Development and Applications , 2008 .

[16]  What CDM Can Tell About What Students Have Learned: An Analysis of TIMSS Eighth Grade Mathematics , 2015 .

[17]  Chia-Yi Chiu,et al.  A General Method of Empirical Q-matrix Validation , 2016, Psychometrika.

[18]  Hoi K. Suen,et al.  Constructing and Validating a Q-Matrix for Cognitive Diagnostic Analyses of a Reading Test , 2013 .

[19]  J. D. L. Torre,et al.  The Generalized DINA Model Framework. , 2011 .

[20]  Wenyi Wang,et al.  An EM-Based Method for Q-Matrix Validation , 2018, Applied psychological measurement.

[21]  K. Tatsuoka.  RULE SPACE: AN APPROACH FOR DEALING WITH MISCONCEPTIONS BASED ON ITEM RESPONSE THEORY , 1983 .

[22]  Jingchen Liu,et al.  Hypothesis Testing of the Q-matrix , 2018, Psychometrika.

[23]  Chia-Yi Chiu,et al.  On the Consistency of Q-Matrix Estimation: A Rejoinder , 2017, Psychometrika.

[24]  Filip Lievens,et al.  Validity and Reliability of Situational Judgement Test Scores , 2016 .

[25]  John T. Willse,et al.  Defining a Family of Cognitive Diagnosis Models Using Log-Linear Models with Latent Variables , 2009 .

[26]  Louis Roussos,et al.  THE FUSION MODEL FOR SKILLS DIAGNOSIS: BLENDING THEORY WITH PRACTICALITY , 2008 .

[27]  Jimmy de la Torre,et al.  A General Cognitive Diagnosis Model for Expert-Defined Polytomous Attributes , 2013 .

[28]  Ren Liu,et al.  The Impact of Q-Matrix Designs on Diagnostic Classification Accuracy in the Presence of Attribute Hierarchies , 2017, Educational and psychological measurement.

[29]  William D. Schafer,et al.  measurement and evaluation in counseling and Development , 2013 .

[30]  Julio Olea,et al.  Inferential Item-Fit Evaluation in Cognitive Diagnosis Modeling , 2017, Applied psychological measurement.

[31]  B. Junker,et al.  Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory , 2001 .

[32]  J. Templin,et al.  The Effects of Q-Matrix Misspecification on Parameter Estimates and Classification Accuracy in the DINA Model , 2008 .

[33]  Hamdollah Ravand,et al.  Application of a Cognitive Diagnostic Model to a High-Stakes Reading Comprehension Test , 2016 .

[34]  Jinsong Chen.  A Residual-Based Approach to Validate Q-Matrix Specifications , 2017, Applied psychological measurement.

[35]  Andrew Izsák,et al.  Diagnosing Teachers’ Understandings of Rational Numbers: Building a Multidimensional Test Within the Diagnostic Classification Framework , 2014 .

[36]  Jeffrey A Douglas,et al.  Higher-order latent trait models for cognitive diagnosis , 2004 .

[37]  Matthew J Madison,et al.  The Effects of Q-Matrix Design on Classification Accuracy in the Log-Linear Cognitive Diagnosis Model , 2015, Educational and psychological measurement.

[38]  Jimmy de la Torre.  The Generalized DINA Model Framework , 2011 .

[39]  Julio Olea,et al.  Application of cognitive diagnosis models to competency-based situational judgment tests. , 2014, Psicothema.

[40]  Jimmy de la Torre,et al.  Cognitively Diagnostic Assessments and the Cognitive Diagnosis Model Framework , 2014 .

[41]  B. Junker,et al.  Cognitive Assessment Models with Few Assumptions , and Connections with Nonparametric IRT , 2001 .

[42]  F. Varadi,et al.  Distinguishing neurocognitive functions in schizophrenia using partially ordered classification models. , 2006, Schizophrenia bulletin.

[43]  Edward Haertel,et al.  An Application of Latent Class Models to Assessment Data , 1984 .