Impact of diagnosticity on the adequacy of models for cognitive diagnosis under a linear attribute structure: A simulation study

Compared to unidimensional item response models (IRMs), cognitive diagnostic models (CDMs) based on latent classes represent examinees' knowledge and item requirements using discrete structures. This study systematically examines the viability of retrofitting CDMs to IRM-based data with a linear attribute structure. The study utilizes a procedure to make the IRM and CDM frameworks comparable and investigates how estimation accuracy is affected by test diagnosticity and the match between the true and fitted models. The study shows that comparable results can be obtained when highly diagnostic IRM data are retrofitted with CDM, and vice versa, retrofitting CDMs to IRM-based data in some conditions can result in considerable examinee misclassification, and model fit indices provide limited indication of the accuracy of item parameter estimation and attribute classification.

[1]  C. Mitchell Dayton,et al.  The Use of Probabilistic Models in the Assessment of Mastery , 1977 .

[2]  B. Junker.  Some statistical models and computational methods that may be useful for cognitively-relevant assessment , 1999 .

[3]  D. Kuhn.  Why development does (and does not) occur : Evidence from the domain of inductive reasoning , 2001 .

[4]  Mark Wilson,et al.  Constructing Measures: An Item Response Modeling Approach , 2004 .

[5]  Joanna S. Gorin,et al.  Item Diffficulty Modeling of Paragraph Comprehension Items , 2006 .

[6]  J. Templin,et al.  Skills Diagnosis Using IRT-Based Latent Class Models , 2007 .

[7]  J. D. L. Torre,et al.  DINA Model and Parameter Estimation: A Didactic , 2009 .

[8]  James E. Corter,et al.  Patterns of Diagnosed Mathematical Content and Process Skills in TIMSS-R Across a Sample of 20 Countries , 2004 .

[9]  Louis V. DiBello,et al.  Guest Editors' Introduction and Overview: IRT‐Based Cognitive Diagnostic Models and Related Methods , 2007 .

[10]  K. Tatsuoka.  RULE SPACE: AN APPROACH FOR DEALING WITH MISCONCEPTIONS BASED ON ITEM RESPONSE THEORY , 1983 .

[11]  Menucha Birenbaum,et al.  Large‐scale diagnostic assessment: comparison of eighth graders’ mathematics performance in the United States, Singapore and Israel , 2005 .

[12]  Jimmy de la Torre,et al.  Model Evaluation and Multiple Strategies in Cognitive Diagnosis: An Analysis of Fraction Subtraction Data , 2008 .

[13]  Mark J. Gierl,et al.  The Attribute Hierarchy Method for Cognitive Assessment: A Variation on Tatsuoka's Rule-Space Approach , 2004 .

[14]  William Stout,et al.  Skills Diagnosis Using IRT‐Based Continuous Latent Trait Models , 2007 .

[15]  E. Maris.  Estimating multiple classification latent class models , 1999 .

[16]  J. Piaget,et al.  The Growth Of Logical Thinking From Childhood To Adolescence: An Essay On The Construction Of Formal Operational Structures , 1958 .

[17]  Edward H. Haertel.  Continuous and discrete latent structure models for item response data , 1990 .

[18]  E. Erikson.  Childhood and Society , 1965 .

[19]  B. Junker,et al.  Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory , 2001 .

[20]  Georg Rasch,et al.  Probabilistic Models for Some Intelligence and Attainment Tests , 1981, The SAGE Encyclopedia of Research Design.

[21]  Jean-Claude Falmagne,et al.  Knowledge spaces , 1998 .

[22]  James W. Pellegrino,et al.  Chapter 9 : Addressing the “Two Disciplines” Problem: Linking Theories of Cognition and Learning With Assessment and Instructional Practice , 1999 .

[23]  Benjamin S. Bloom,et al.  A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives , 2000 .