Incorporating Student Covariates in Cognitive Diagnosis Models

In educational measurement, cognitive diagnosis models have been developed to allow assessment of specific skills that are needed to perform tasks. Skill knowledge is characterized as present or absent and represented by a vector of binary indicators, or the skill set profile. After determining which skills are needed for each assessment item, a model is specified for the relationship between item responses and skill set profiles. Cognitive diagnosis models are often used for diagnosis, that is, for classifying students into the different skill set profiles. Generally, cognitive diagnosis models do not exploit student covariate information. However, investigating the effects of student covariates, such as gender, SES, or educational interventions, on skill knowledge mastery is important in education research, and covariate information may improve classification of students to skill set profiles. We extend a common cognitive diagnosis model, the DINA model, by modeling the relationship between the latent skill knowledge indicators and covariates. The probability of skill mastery is modeled as a logistic regression model, possibly with a student-level random intercept, giving a higher-order DINA model with a latent regression. Simulations show that parameter recovery is good for these models and that inclusion of covariates can improve skill diagnosis. When applying our methods to data from an online tutor, we obtain reasonable and interpretable parameter estimates that allow more detailed characterization of groups of students who differ in their predicted skill set profiles.

[1]  Wenmin Zhang.  Detecting differential item functioning using the DINA model , 2006 .

[2]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[3]  K. Jöreskog.  Simultaneous factor analysis in several populations , 1971 .

[4]  Jonathan Templin,et al.  Diagnostic Measurement: Theory, Methods, and Applications , 2010 .

[5]  E. Muraki,et al.  Chapter 3: Scaling Procedures in NAEP , 1992 .

[6]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[7]  Henk Kelderman,et al.  The Mixed Birnbaum Model: Estimation using Collateral Information , 2000 .

[8]  J. Templin,et al.  The Effects of Q-Matrix Misspecification on Parameter Estimates and Classification Accuracy in the DINA Model , 2008 .

[9]  J. D. L. Torre,et al.  DINA Model and Parameter Estimation: A Didactic , 2009 .

[10]  K. Larsen,et al.  Interpreting Parameters in the Logistic Regression Model with Random Effects , 2000, Biometrics.

[11]  Aeilko H. Zwinderman,et al.  A generalized rasch model for manifest predictors , 1991 .

[12]  George B. Macready,et al.  Concomitant-Variable Latent-Class Models , 1988 .

[13]  Rebecca Nugent,et al.  A Comparison of Student Skill Knowledge Estimates , 2009, EDM.

[14]  A. Goldberger,et al.  Estimation of a Model with Multiple Indicators and Multiple Causes of a Single Latent Variable , 1975 .

[15]  Howard T. Everson,et al.  Methodology Review: Statistical Approaches for Assessing Measurement Bias , 1993 .

[16]  Jay Magidson,et al.  Latent Class Factor and Cluster Models, Bi-Plots, and Related Graphical Displays , 2001 .

[17]  Jeffrey A Douglas,et al.  Higher-order latent trait models for cognitive diagnosis , 2004 .

[18]  Eugene G. Johnson,et al.  Scaling Procedures in NAEP , 1992 .

[19]  K. Tatsuoka.  RULE SPACE: AN APPROACH FOR DEALING WITH MISCONCEPTIONS BASED ON ITEM RESPONSE THEORY , 1983 .

[20]  A. Formann.  Linear Logistic Latent Class Analysis for Polytomous Data , 1992 .

[21]  John A. Self,et al.  Model-based cognitive diagnosis , 1993, User Modeling and User-Adapted Interaction.

[22]  Chia-Yi Chiu,et al.  Cluster Analysis for Cognitive Diagnosis: Theory and Applications , 2009 .

[23]  E. Maris.  Estimating multiple classification latent class models , 1999 .

[24]  W. Meredith.  Measurement invariance, factor analysis and factorial invariance , 1993 .

[25]  Edward H. Haertel.  Using restricted latent class models to map the skill structure of achievement items , 1989 .

[26]  Howard Wainer,et al.  Use of item response theory in the study of group differences in trace lines. , 1988 .

[27]  Robert J. Mislevy.  Exploiting auxiliary information about examinees in the estimation of item parameters , 1986 .

[28]  Allan S. Cohen,et al.  A Multilevel Mixture IRT Model With an Application to DIF , 2010 .

[29]  Tiffany Barnes,et al.  The q-matrix method of fault-tolerant teaching in knowledge assessment and data mining , 2003 .

[30]  Jeroen K. Vermunt,et al.  7. Multilevel Latent Class Models , 2003 .

[31]  Michel Wedel,et al.  Concomitant variables in finite mixture models , 2002 .

[32]  H. van der Flier,et al.  Collateral information and Mixed Rasch models , 1999 .

[33]  Elizabeth Ayers,et al.  IRT Modeling of Tutor Performance to Predict End-of-Year Exam Scores , 2008 .

[34]  Matthias von Davier,et al.  FITTING THE STRUCTURED GENERAL DIAGNOSTIC MODEL TO NAEP DATA , 2008 .

[35]  L. A. Goodman,et al.  Latent Structure Analysis of a Set of Multidimensional Contingency Tables , 1984 .

[36]  M. Davier.  Hierarchical mixtures of diagnostic models , 2010 .

[37]  Robert J. Mislevy,et al.  Estimation of Latent Group Effects , 1985 .

[38]  S. Embretson.  A general latent trait model for response processes , 1984 .

[39]  B. Junker,et al.  Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory , 2001 .

[40]  Jonathan Templin,et al.  Robustness of Hierarchical Modeling of Skill Association in Cognitive Diagnosis Models , 2008 .

[41]  Sarah M. Hartz,et al.  A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. , 2002 .

[42]  D. Rubin.  Multiple imputation for nonresponse in surveys , 1989 .

[43]  C. Mitchell Dayton,et al.  The Use of Probabilistic Models in the Assessment of Mastery , 1977 .

[44]  Bengt Muthén,et al.  Multiple Group IRT Modeling: Applications to Item Bias Analysis , 1985 .