Evaluation of Model Fit in Cognitive Diagnosis Models

Cognitive diagnosis models (CDMs) estimate student ability profiles using latent attributes. Model fit to the data needs to be ascertained in order to determine whether inferences from CDMs are valid. This study investigated the usefulness of some popular model fit statistics to detect CDM fit including relative fit indices (AIC, BIC, and CAIC), and absolute fit indices (RMSEA2, ABS(fcor) and MAX(χ2jj′)). These fit indices were assessed under different CDM settings with respect to Q-matrix misspecification and CDM misspecification. Results showed that relative fit indices selected the correct DINA model most of the times and selected the correct G-DINA model well across most conditions. Absolute fit indices rejected the true DINA model if the Q-matrix was misspecified in any way. Absolute fit indices rejected the true G-DINA model whenever the Q-matrix was under-specified. RMSEA2 could be artificially low when the Q-matrix was over-specified.

[1]  H. Bozdogan.  Model selection and Akaike's Information Criterion (AIC): The general theory and its analytical extensions , 1987 .

[2]  Laine Bradshaw,et al.  Hierarchical Diagnostic Classification Models: A Family of Models for Estimating and Testing Attribute Hierarchies , 2014, Psychometrika.

[3]  Jimmy de la Torre,et al.  Model Evaluation and Multiple Strategies in Cognitive Diagnosis: An Analysis of Fraction Subtraction Data , 2008 .

[4]  Jonathan Templin,et al.  Diagnostic Measurement: Theory, Methods, and Applications , 2010 .

[5]  Jimmy de la Torre,et al.  Relative and Absolute Fit Evaluation in Cognitive Diagnosis Modeling. , 2013 .

[6]  Matthias von Davier,et al.  A General Diagnostic Model Applied to Language Testing Data. Research Report. ETS RR-05-16. , 2005 .

[7]  B. Junker,et al.  Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory , 2001 .

[8]  Sarah M. Hartz,et al.  A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. , 2002 .

[9]  Jimmy de la Torre,et al.  An Empirically Based Method of Q‐Matrix Validation for the DINA Model: Development and Applications , 2008 .

[10]  L. T. DeCarlo.  On the Analysis of Fraction Subtraction Data: The DINA Model, Classification, Latent Class Sizes, and the Q-Matrix , 2011 .

[11]  Ying Liu,et al.  Testing Person Fit in Cognitive Diagnosis , 2009 .

[12]  L. Thelma.  Meeting of the National Council on Measurement in Education , 2000 .

[13]  T. Başokçu.  Classification Accuracy Effects of Q-Matrix Validation and Sample Size in DINA and G-DINA Models , 2014 .

[14]  André A. Rupp,et al.  The Impact of Model Misspecification on Parameter Estimation and Item‐Fit Assessment in Log‐Linear Diagnostic Classification Models , 2012 .

[15]  Ding Shuliang,et al.  Test Construction for Cognitive Diagnosis , 2010, 2010 Asia-Pacific Conference on Wearable Computing Systems.

[16]  J. Templin,et al.  The Effects of Q-Matrix Misspecification on Parameter Estimates and Classification Accuracy in the DINA Model , 2008 .

[17]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[18]  J. Templin,et al.  Measurement of psychological disorders using cognitive diagnosis models. , 2006, Psychological methods.

[19]  K. Tatsuoka.  RULE SPACE: AN APPROACH FOR DEALING WITH MISCONCEPTIONS BASED ON ITEM RESPONSE THEORY , 1983 .

[20]  Andrew Izsák,et al.  Diagnosing Teachers’ Understandings of Rational Numbers: Building a Multidimensional Test Within the Diagnostic Classification Framework , 2014 .

[21]  Young-sun Lee,et al.  A Cognitive Diagnostic Modeling of Attribute Mastery in Massachusetts, Minnesota, and the U.S. National Sample Using the TIMSS 2007 , 2011 .

[22]  H. Akaike.  A new look at the statistical model identification , 1974 .

[23]  Jeffrey A Douglas,et al.  Higher-order latent trait models for cognitive diagnosis , 2004 .

[24]  Jonathan Templin,et al.  Using Efficient Model Based Sum‐Scores for Conducting Skills Diagnoses , 2007 .

[25]  J. D. L. Torre,et al.  The Generalized DINA Model Framework. , 2011 .

[26]  John T. Willse,et al.  Defining a Family of Cognitive Diagnosis Models Using Log-Linear Models with Latent Variables , 2009 .

[27]  Harry Joe,et al.  Assessing Approximate Fit in Categorical Data Analysis , 2014, Multivariate behavioral research.

[28]  Matthias von Davier,et al.  A general diagnostic model applied to language testing data. , 2008, The British journal of mathematical and statistical psychology.

[29]  J. D. L. Torre,et al.  A Note on the Invariance of the DINA Model Parameters. , 2010 .

[30]  L. T. DeCarlo.  Recognizing Uncertainty in the Q-Matrix via a Bayesian Extension of the DINA Model , 2012 .