Assessing Fit of Cognitive Diagnostic Models A Case Study

A cognitive diagnostic model uses information from educational experts to describe the relationships between item performances and posited proficiencies. When the cognitive relationships can be described using a fully Bayesian model, Bayesian model checking procedures become available. Checking models tied to cognitive theory of the domains provides feedback to educators about the underlying cognitive theory. This article suggests a number of graphics and statistics for diagnosing problems with cognitive diagnostic models expressed as Bayesian networks. The suggested diagnostics allow the authors to identify the inadequacy of an earlier cognitive diagnostic model and to hypothesize an improved model that provides better fit to the data.

[1]  K. Tatsuoka.  Toward an Integration of Item-Response Theory and Cognitive Error Diagnosis. , 1987 .

[2]  Mary F. Klein.  Logical Error Analysis and Construction of Tests to Diagnose Student "Bugs" in Addition and Subtraction of Fractions. , 1981 .

[3]  B. Junker,et al.  Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory , 2001 .

[4]  E. Maris.  Estimating multiple classification latent class models , 1999 .

[5]  R. Almond,et al.  Focus Article: On the Structure of Educational Assessments , 2003 .

[6]  Kikumi K. Tatsuoka,et al.  Differential Item Functioning Resulting From The Use of Different Solution Strategies , 1988 .

[7]  Robert J. Mislevy,et al.  Specifying and Refining a Measurement Model for a Computer-Based Interactive Assessment , 2004 .

[8]  Russell G. Almond,et al.  Bayes Nets in Educational Assessment: Where the Numbers Come From , 1999, UAI.

[9]  Jeffrey A Douglas,et al.  Higher-order latent trait models for cognitive diagnosis , 2004 .

[10]  R. Mislevy.  Evidence and inference in educational assessment , 1994 .

[11]  Matthias von Davier,et al.  A GENERAL DIAGNOSTIC MODEL APPLIED TO LANGUAGE TESTING DATA , 2005 .

[12]  Robert J. Mislevy,et al.  Specifying and Refining a Measurement Model for a Simulation-Based Assessment. CSE Report 619. , 2004 .

[13]  K. Tatsuoka.  RULE SPACE: AN APPROACH FOR DEALING WITH MISCONCEPTIONS BASED ON ITEM RESPONSE THEORY , 1983 .

[14]  Walter R. Gilks,et al.  BUGS - Bayesian inference Using Gibbs Sampling Version 0.50 , 1995 .

[15]  Russell G. Almond,et al.  Models for Conditional Probability Tables in Educational Assessment , 2001, AISTATS.

[16]  Edward H. Haertel.  Using restricted latent class models to map the skill structure of achievement items , 1989 .

[17]  R. Hambleton,et al.  Item Response Theory , 1984, The History of Educational Measurement.

[18]  Robert J. Mislevy,et al.  PROBABILITY‐BASED INFERENCE IN COGNITIVE DIAGNOSIS , 1994 .

[19]  K. Chaloner,et al.  A Bayesian approach to outlier detection and residual analysis , 1988 .

[20]  Bradley P. Carlin,et al.  Bayesian measures of model complexity and fit , 2002 .

[21]  Russell G. Almond,et al.  Graphical Models and Computerized Adaptive Testing , 1998 .

[22]  R. Hambleton,et al.  ANALYSIS OF EMPIRICAL DATA USING TWO LOGISTIC LATENT TRAIT MODELS , 1973 .

[23]  Russell G. Almond,et al.  ASSESSING FIT OF MODELS WITH DISCRETE PROFICIENCY VARIABLES IN EDUCATIONAL ASSESSMENT , 2004 .

[24]  A. Hasman,et al.  Probabilistic reasoning in intelligent systems: Networks of plausible inference , 1991 .

[25]  S. Chib,et al.  Bayesian residual analysis for binary response regression models , 1995 .