The Impact of Q-Matrix Designs on Diagnostic Classification Accuracy in the Presence of Attribute Hierarchies

There is an increasing demand for assessments that can provide more fine-grained information about examinees. In response to the demand, diagnostic measurement provides students with feedback on their strengths and weaknesses on specific skills by classifying them into mastery or nonmastery attribute categories. These attributes often form a hierarchical structure because student learning and development is a sequential process where many skills build on others. However, it remains to be seen if we can use information from the attribute structure and work that into the design of the diagnostic tests. The purpose of this study is to introduce three approaches of Q-matrix design and investigate their impact on classification results under different attribute structures. Results indicate that the adjacent approach provides higher accuracy in a shorter test length when compared with other Q-matrix design approaches. This study provides researchers and practitioners guidance on how to design the Q-matrix in diagnostic tests, which are in high demand from educators.

[1]  J. Templin,et al.  Measurement of psychological disorders using cognitive diagnosis models. , 2006, Psychological methods.

[2]  Mark Gierl,et al.  Estimating Classification Consistency and Accuracy for Cognitive Diagnostic Assessment. , 2012 .

[3]  Håkan Hult,et al.  From senior student to novice worker: learning trajectories in political science, psychology and mechanical engineering , 2006 .

[4]  Laine Bradshaw,et al.  Combining Item Response Theory and Diagnostic Classification Models: A Psychometric Model for Scaling Ability and Diagnosing Misconceptions , 2014, Psychometrika.

[5]  Mark J. Gierl,et al.  Cognitive Diagnostic Assessment for Education: Using the Attribute Hierarchy Method to Make Diagnostic Inferences About Examinees' Cognitive Skills , 2007 .

[6]  Jimmy de la Torre,et al.  An Empirically Based Method of Q‐Matrix Validation for the DINA Model: Development and Applications , 2008 .

[7]  Chia-Yi Chiu,et al.  Cluster Analysis for Cognitive Diagnosis: Theory and Applications , 2009 .

[8]  Anne Corinne Huggins-Manley,et al.  The Specification of Attribute Structures and Its Effects on Classification Accuracy in Diagnostic Test Design , 2016 .

[9]  Athanassios Jimoyiannis,et al.  Computer simulations in physics teaching and learning: a case study on students' understanding of trajectory motion , 2001, Comput. Educ..

[10]  J. D. L. Torre,et al.  DINA Model and Parameter Estimation: A Didactic , 2009 .

[11]  Laine Bradshaw,et al.  Hierarchical Diagnostic Classification Models: A Family of Models for Estimating and Testing Attribute Hierarchies , 2014, Psychometrika.

[12]  Chia-Yi Chiu,et al.  A Nonparametric Approach to Cognitive Diagnosis by Proximity to Ideal Response Patterns , 2013, J. Classif..

[13]  Russell G. Almond,et al.  Assessing Fit of Cognitive Diagnostic Models A Case Study , 2007 .

[14]  Mark J. Gierl,et al.  The Attribute Hierarchy Method for Cognitive Assessment: A Variation on Tatsuoka's Rule-Space Approach , 2004 .

[15]  J. Templin,et al.  The Effects of Q-Matrix Misspecification on Parameter Estimates and Classification Accuracy in the DINA Model , 2008 .

[16]  Chia-Yi Chiu,et al.  A General Method of Empirical Q-matrix Validation , 2016, Psychometrika.

[17]  John T. Willse,et al.  Defining a Family of Cognitive Diagnosis Models Using Log-Linear Models with Latent Variables , 2009 .

[18]  L. T. DeCarlo.  On the Analysis of Fraction Subtraction Data: The DINA Model, Classification, Latent Class Sizes, and the Q-Matrix , 2011 .

[19]  Gongjun Xu,et al.  Identifiability of Diagnostic Classification Models , 2015, Psychometrika.

[20]  Laine Bradshaw,et al.  Measuring the Reliability of Diagnostic Classification Model Examinee Estimates , 2013, J. Classif..

[21]  K. Tatsuoka.  Cognitive Assessment: An Introduction to the Rule Space Method , 2009 .

[22]  Jinxiang Hu,et al.  Evaluation of Model Fit in Cognitive Diagnosis Models , 2016 .

[23]  Patrick McClure,et al.  The Impact Model. , 2006 .

[24]  Edward H. Haertel.  Using restricted latent class models to map the skill structure of achievement items , 1989 .

[25]  Matthew J Madison,et al.  The Effects of Q-Matrix Design on Classification Accuracy in the Log-Linear Cognitive Diagnosis Model , 2015, Educational and psychological measurement.

[26]  Denny Borsboom,et al.  Test Validity in Cognitive Assessment , 2007 .

[27]  S. Haberman,et al.  An NCME Instructional Module on Subscores , 2011 .

[28]  André A. Rupp,et al.  The Impact of Model Misspecification on Parameter Estimation and Item‐Fit Assessment in Log‐Linear Diagnostic Classification Models , 2012 .

[29]  Martin A. Simon,et al.  Explicating the Role of Mathematical Tasks in Conceptual Learning: An Elaboration of the Hypothetical Learning Trajectory , 2004 .

[30]  K. Tatsuoka.  RULE SPACE: AN APPROACH FOR DEALING WITH MISCONCEPTIONS BASED ON ITEM RESPONSE THEORY , 1983 .

[31]  Chia-Yi Chiu.  Statistical Refinement of the Q-Matrix in Cognitive Diagnosis , 2013 .