The Effects of Q-Matrix Design on Classification Accuracy in the Log-Linear Cognitive Diagnosis Model

Diagnostic classification models are psychometric models that aim to classify examinees according to their mastery or non-mastery of specified latent characteristics. These models are well-suited for providing diagnostic feedback on educational assessments because of their practical efficiency and increased reliability when compared with other multidimensional measurement models. A priori specifications of which latent characteristics or attributes are measured by each item are a core element of the diagnostic assessment design. This item–attribute alignment, expressed in a Q-matrix, precedes and supports any inference resulting from the application of the diagnostic classification model. This study investigates the effects of Q-matrix design on classification accuracy for the log-linear cognitive diagnosis model. Results indicate that classification accuracy, reliability, and convergence rates improve when the Q-matrix contains isolated information from each measured attribute.

[1]  Denny Borsboom,et al.  Test Validity in Cognitive Assessment , 2007 .

[2]  R. P. McDonald,et al.  Test Theory: A Unified Treatment , 1999 .

[3]  S. Haberman,et al.  An NCME Instructional Module on Subscores , 2011 .

[4]  K. Tatsuoka.  Toward an Integration of Item-Response Theory and Cognitive Error Diagnosis. , 1987 .

[5]  Laine Bradshaw,et al.  Measuring the Reliability of Diagnostic Classification Model Examinee Estimates , 2013, J. Classif..

[6]  J. Templin,et al.  Measurement of psychological disorders using cognitive diagnosis models. , 2006, Psychological methods.

[7]  Mark Gierl,et al.  Estimating Classification Consistency and Accuracy for Cognitive Diagnostic Assessment. , 2012 .

[8]  J. Templin,et al.  The Effects of Q-Matrix Misspecification on Parameter Estimates and Classification Accuracy in the DINA Model , 2008 .

[9]  André A. Rupp,et al.  The Impact of Model Misspecification on Parameter Estimation and Item‐Fit Assessment in Log‐Linear Diagnostic Classification Models , 2012 .

[10]  K. Tatsuoka.  RULE SPACE: AN APPROACH FOR DEALING WITH MISCONCEPTIONS BASED ON ITEM RESPONSE THEORY , 1983 .

[11]  Andrew Izsák,et al.  Diagnosing Teachers’ Understandings of Rational Numbers: Building a Multidimensional Test Within the Diagnostic Classification Framework , 2014 .

[12]  John T. Willse,et al.  Defining a Family of Cognitive Diagnosis Models Using Log-Linear Models with Latent Variables , 2009 .

[13]  Laine Bradshaw,et al.  Combining Item Response Theory and Diagnostic Classification Models: A Psychometric Model for Scaling Ability and Diagnosing Misconceptions , 2014, Psychometrika.

[14]  Ding Shuliang,et al.  Test Construction for Cognitive Diagnosis , 2010, 2010 Asia-Pacific Conference on Wearable Computing Systems.

[15]  Chia-Yi Chiu,et al.  Cluster Analysis for Cognitive Diagnosis: Theory and Applications , 2009 .

[16]  Kristen Huff,et al.  Cognitive Diagnostic Assessment for Education: The Demand for Cognitive Diagnostic Assessment , 2007 .

[17]  Edward H. Haertel.  Using restricted latent class models to map the skill structure of achievement items , 1989 .

[18]  Jonathan Templin,et al.  Diagnostic Measurement: Theory, Methods, and Applications , 2010 .

[19]  L. T. DeCarlo.  On the Analysis of Fraction Subtraction Data: The DINA Model, Classification, Latent Class Sizes, and the Q-Matrix , 2011 .

[20]  B. Junker,et al.  Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory , 2001 .

[21]  J. D. L. Torre,et al.  DINA Model and Parameter Estimation: A Didactic , 2009 .