Hypothesis Testing of the Q-matrix

The recent surge of interests in cognitive assessment has led to the development of cognitive diagnosis models. Central to many such models is a specification of the Q-matrix, which relates items to latent attributes that have natural interpretations. In practice, the Q-matrix is usually constructed subjectively by the test designers. This could lead to misspecification, which could result in lack of fit of the underlying statistical model. To test possible misspecification of the Q-matrix, traditional goodness of fit tests, such as the Chi-square test and the likelihood ratio test, may not be applied straightforwardly due to the large number of possible response patterns. To address this problem, this paper proposes a new statistical method to test the goodness fit of the Q-matrix, by constructing test statistics that measure the consistency between a provisional Q-matrix and the observed data for a general family of cognitive diagnosis models. Limiting distributions of the test statistics are derived under the null hypothesis that can be used for obtaining the test p-values. Simulation studies as well as a real data example are presented to demonstrate the usefulness of the proposed method.

[1]  William Stout,et al.  Skills Diagnosis Using IRT‐Based Continuous Latent Trait Models , 2007 .

[2]  Chia-Yi Chiu.  Statistical Refinement of the Q-Matrix in Cognitive Diagnosis , 2013 .

[3]  Jingchen Liu,et al.  Data-Driven Learning of Q-Matrix , 2012, Applied psychological measurement.

[4]  Z. Ying,et al.  Statistical Analysis of Q-Matrix Based Diagnostic Classification Models , 2015, Journal of the American Statistical Association.

[5]  John T. Willse,et al.  Defining a Family of Cognitive Diagnosis Models Using Log-Linear Models with Latent Variables , 2009 .

[6]  L. T. DeCarlo.  On the Analysis of Fraction Subtraction Data: The DINA Model, Classification, Latent Class Sizes, and the Q-Matrix , 2011 .

[7]  Curtis Tatsuoka,et al.  Data analytic methods for latent partially ordered classification models , 2002 .

[8]  J. D. L. Torre,et al.  The Generalized DINA Model Framework. , 2011 .

[9]  Jingchen Liu,et al.  Theory of the Self-learning Q-Matrix. , 2010, Bernoulli : official journal of the Bernoulli Society for Mathematical Statistics and Probability.

[10]  K. Tatsuoka.  Toward an Integration of Item-Response Theory and Cognitive Error Diagnosis. , 1987 .

[11]  Gongjun Xu,et al.  Identifying Latent Structures in Restricted Latent Class Models , 2018, Journal of the American Statistical Association.

[12]  Albert Maydeu-Olivares,et al.  Limited information estimation and testing of Thurstonian models for paired comparison data under multiple judgment sampling , 2001 .

[13]  Jimmy de la Torre,et al.  An Empirically Based Method of Q‐Matrix Validation for the DINA Model: Development and Applications , 2008 .

[14]  Kikumi K. Tatsuoka,et al.  A Probabilistic Model for Diagnosing Misconceptions By The Pattern Classification Approach , 1985 .

[15]  André A. Rupp,et al.  Feature Selection for Choosing and Assembling Measurement Models: A Building-Block-Based Organization , 2002 .

[16]  Jeffrey A Douglas,et al.  Higher-order latent trait models for cognitive diagnosis , 2004 .

[17]  E. Lehmann.  Testing Statistical Hypotheses , 1960 .

[18]  Mark J. Gierl,et al.  The Attribute Hierarchy Method for Cognitive Assessment: A Variation on Tatsuoka's Rule-Space Approach , 2004 .

[19]  J. Templin,et al.  The Effects of Q-Matrix Misspecification on Parameter Estimates and Classification Accuracy in the DINA Model , 2008 .

[20]  Gongjun Xu,et al.  Identifiability of restricted latent class models with binary responses , 2016, 1603.04140.

[21]  J. Templin,et al.  Measurement of psychological disorders using cognitive diagnosis models. , 2006, Psychological methods.

[22]  Ab Mooijaart,et al.  Type I errors and power of the parametric bootstrap goodness-of-fit test: full and limited information. , 2003, The British journal of mathematical and statistical psychology.

[23]  Bodhisattva Sen,et al.  Model based bootstrap methods for interval censored data , 2013, Comput. Stat. Data Anal..

[24]  Jonathan Templin,et al.  Diagnostic Measurement: Theory, Methods, and Applications , 2010 .

[25]  Matthias von Davier,et al.  A General Diagnostic Model Applied to Language Testing Data. Research Report. ETS RR-05-16. , 2005 .

[26]  Chia-Yi Chiu,et al.  A General Method of Empirical Q-matrix Validation , 2016, Psychometrika.

[27]  K. Tatsuoka.  Cognitive Assessment: An Introduction to the Rule Space Method , 2009 .

[28]  J. Templin,et al.  Skills Diagnosis Using IRT-Based Latent Class Models , 2007 .

[29]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[30]  David J. Bartholomew,et al.  The Goodness of Fit of Latent Trait Models in Attitude Measurement , 1999 .

[31]  Curtis Tatsuoka.  Corrigendum: Data analytic methods for latent partially ordered classification models , 2005 .

[32]  Albert Maydeu-Olivares,et al.  Limited- and Full-Information Estimation and Goodness-of-Fit Testing in 2n Contingency Tables , 2005 .

[33]  D. Thissen,et al.  Limited-information goodness-of-fit testing of item response theory models for sparse 2 tables. , 2006, The British journal of mathematical and statistical psychology.

[34]  B. Sen,et al.  Inconsistency of bootstrap: The Grenander estimator , 2010, 1010.3825.

[35]  Zhiliang Ying,et al.  Non-identifiability, equivalence classes, and attribute-specific classification in Q-matrix based Cognitive Diagnosis Models , 2013 .

[36]  Chia-Yi Chiu,et al.  Cluster Analysis for Cognitive Diagnosis: Theory and Applications , 2009 .

[37]  J. Templin,et al.  Unique Characteristics of Diagnostic Classification Models: A Comprehensive Review of the Current State-of-the-Art , 2008 .

[38]  B. Junker,et al.  Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory , 2001 .

[39]  Sarah M. Hartz,et al.  A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. , 2002 .

[40]  L. T. DeCarlo.  Recognizing Uncertainty in the Q-Matrix via a Bayesian Extension of the DINA Model , 2012 .