Assessing Item-Level Fit for the DINA Model

This research focuses on developing item-level fit checking procedures in the context of diagnostic classification models (DCMs), and more specifically for the “Deterministic Input; Noisy ‘And’ gate” (DINA) model. Although there is a growing body of literature discussing model fit checking methods for DCM, the item-level fit analysis is not adequately discussed in literature. This study intends to take an initiative to fill in this gap. Two approaches are proposed, one stems from classical goodness-of-fit test statistics coupled with the Expectation-Maximization algorithm for model estimation, and the other is the posterior predictive model checking (PPMC) method coupled with the Markov chain Monte Carlo estimation. For both approaches, the chi-square statistic and a power-divergence index are considered, along with Stone’s method for considering uncertainty in latent attribute estimation. A simulation study with varying manipulated factors is carried out. Results show that both approaches are promising if Stone’s method is imposed, but the classical goodness-of-fit approach has a much higher detection rate (i.e., proportion of misfit items that are correctly detected) than the PPMC method.

[1]  Laine Bradshaw,et al.  Hierarchical Diagnostic Classification Models: A Family of Models for Estimating and Testing Attribute Hierarchies , 2014, Psychometrika.

[2]  Jim Albert,et al.  Discrepancy measures for item fit analysis in item response theory , 2011 .

[3]  D. Thissen,et al.  Likelihood-Based Item-Fit Indices for Dichotomous Item Response Theory Models , 2000 .

[4]  Jingchen Liu,et al.  Data-Driven Learning of Q-Matrix , 2012, Applied psychological measurement.

[5]  Edward H. Haertel.  Using restricted latent class models to map the skill structure of achievement items , 1989 .

[6]  Louis A. Roussos,et al.  The fusion model skills diagnosis system , 2007 .

[7]  Youn Seon Lim,et al.  Efficient Models for Cognitive Diagnosis With Continuous and Mixed-Type Latent Variables , 2015, Applied psychological measurement.

[8]  J. Templin,et al.  Skills Diagnosis Using IRT-Based Latent Class Models , 2007 .

[9]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[10]  Ying Liu,et al.  Testing Person Fit in Cognitive Diagnosis , 2009 .

[11]  E. Maris.  Estimating multiple classification latent class models , 1999 .

[12]  J. D. L. Torre,et al.  DINA Model and Parameter Estimation: A Didactic , 2009 .

[13]  Ying Cui,et al.  The Hierarchy Consistency Index: Evaluating Person Fit for Cognitive Diagnostic Assessment , 2009 .

[14]  Jonathan Templin,et al.  Robustness of Hierarchical Modeling of Skill Association in Cognitive Diagnosis Models , 2008 .

[15]  Bradley P. Carlin,et al.  Bayesian measures of model complexity and fit , 2002 .

[16]  Chun Wang,et al.  Mutual Information Item Selection Method in Cognitive Diagnostic Computerized Adaptive Testing With Short Test Length , 2013 .

[17]  W. M. Yen.  Using Simulation Results to Choose a Latent Trait Model , 1981 .

[18]  H. Akaike.  A new look at the statistical model identification , 1974 .

[19]  Jeffrey A Douglas,et al.  Higher-order latent trait models for cognitive diagnosis , 2004 .

[20]  W. D. Linden.  Multidimensionality in item response theory , 2013 .

[21]  Craig N. Mills,et al.  A Comparison of Several Goodness-of-Fit Statistics , 1985 .

[22]  J. D. L. Torre,et al.  Evaluating the Wald Test for Item‐Level Comparison of Saturated and Reduced Models in Cognitive Diagnosis , 2013 .

[23]  John T. Willse,et al.  Defining a Family of Cognitive Diagnosis Models Using Log-Linear Models with Latent Variables , 2009 .

[24]  Xiao-Li Meng,et al.  Posterior Predictive $p$-Values , 1994 .

[25]  Matthias von Davier,et al.  Investigation of model fit and score scale comparability in international assessments , 2011 .

[26]  J. D. L. Torre,et al.  The Generalized DINA Model Framework. , 2011 .

[27]  Jonathan Templin,et al.  Diagnostic Measurement: Theory, Methods, and Applications , 2010 .

[28]  Hal S. Stern,et al.  Posterior Predictive Assessment of Item Response Theory Models , 2006 .

[29]  Steven W. Nydick,et al.  Comparing Two Algorithms for Calibrating the Restricted Non-Compensatory Multidimensional IRT Model , 2015, Applied psychological measurement.

[30]  Clement A. Stone.  Monte Carlo Based Null Distribution for an Alternative Goodness‐of‐Fit Test Statistic in IRT Models , 2000 .

[31]  Robert J. Mislevy,et al.  Posterior Predictive Model Checking for Multidimensionality in Item Response Theory , 2006 .

[32]  James M. Robins,et al.  Asymptotic Distribution of P Values in Composite Null Models , 2000 .

[33]  J. Hagenaars.  Loglinear Models with Latent Variables , 1993 .

[34]  Bengt Muthén,et al.  Bayesian structural equation modeling: a more flexible representation of substantive theory. , 2012, Psychological methods.

[35]  Matthias von Davier,et al.  A general diagnostic model applied to language testing data. , 2008, The British journal of mathematical and statistical psychology.

[36]  Hua-Hua Chang,et al.  Restrictive Stochastic Item Selection Methods in Cognitive Diagnostic Computerized Adaptive Testing. , 2011 .

[37]  B. Junker,et al.  Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory , 2001 .

[38]  J. Templin,et al.  The Effects of Q-Matrix Misspecification on Parameter Estimates and Classification Accuracy in the DINA Model , 2008 .

[39]  Sophia Blau,et al.  Goodness Of Fit Statistics For Discrete Multivariate Data , 2016 .

[40]  P L Fidler,et al.  Goodness-of-Fit Testing for Latent Class Models. , 1993, Multivariate behavioral research.

[41]  Howard Eisner,et al.  Information Age , 2008, Wiley Encyclopedia of Computer Science and Engineering.

[42]  N. Hjort,et al.  Post-Processing Posterior Predictive p Values , 2006 .

[43]  Russell G. Almond,et al.  Assessing Fit of Cognitive Diagnostic Models A Case Study , 2007 .

[44]  B. Junker,et al.  Cognitive Assessment Models with Few Assumptions , and Connections with Nonparametric IRT , 2001 .

[45]  André A. Rupp,et al.  The Impact of Model Misspecification on Parameter Estimation and Item‐Fit Assessment in Log‐Linear Diagnostic Classification Models , 2012 .

[46]  Jimmy de la Torre,et al.  Model Evaluation and Multiple Strategies in Cognitive Diagnosis: An Analysis of Fraction Subtraction Data , 2008 .

[47]  Jimmy de la Torre,et al.  Relative and Absolute Fit Evaluation in Cognitive Diagnosis Modeling. , 2013 .

[48]  M. Reckase.  Multidimensional Item Response Theory , 2009 .

[49]  Sandip Sinharay,et al.  Model Diagnostics for Bayesian Networks , 2004 .

[50]  Jimmy de la Torre,et al.  An Empirically Based Method of Q‐Matrix Validation for the DINA Model: Development and Applications , 2008 .