A Nonparametric Approach to Cognitive Diagnosis by Proximity to Ideal Response Patterns

A trend in educational testing is to go beyond unidimensional scoring and provide a more complete profile of skills that have been mastered and those that have not. To achieve this, cognitive diagnosis models have been developed that can be viewed as restricted latent class models. Diagnosis of class membership is the statistical objective of these models. As an alternative to latent class modeling, a nonparametric procedure is introduced that only requires specification of an item-by-attribute association matrix, and classifies according to minimizing a distance measure between observed responses, and the ideal response for a given attribute profile that would be implied by the item-by-attribute association matrix. This procedure requires no statistical parameter estimation, and can be used on a sample size as small as 1. Heuristic arguments are given for why the nonparametric procedure should be effective under various possible cognitive diagnosis models for data generation. Simulation studies compare classification rates with parametric models, and consider a variety of distance measures, data generation models, and the effects of model misspecification. A real data example is provided with an analysis of agreement between the nonparametric method and parametric approaches.

[1]  Chia-Yi Chiu,et al.  Cluster Analysis for Cognitive Diagnosis: Theory and Applications , 2009 .

[2]  Kikumi K. Tatsuoka,et al.  Computerized Cognitive Diagnostic Adaptive Testing: Effect on Remedial Instruction as Empirical Validation , 1997 .

[3]  E. Maris.  Estimating multiple classification latent class models , 1999 .

[4]  Kikumi K. Tatsuoka,et al.  Bug Distribution and Pattern Classification. , 1985 .

[5]  Tiffany Barnes.  Novel Derivation and Application of Skill Matrices: The q-Matrix Method , 2010 .

[6]  Rebecca Nugent,et al.  Skill set profile clustering based on student capability vectors computed from online tutoring data , 2008 .

[7]  J. Templin,et al.  Measurement of psychological disorders using cognitive diagnosis models. , 2006, Psychological methods.

[8]  Kikumi K. Tatsuoka,et al.  A Probabilistic Model for Diagnosing Misconceptions By The Pattern Classification Approach , 1985 .

[9]  Jeffrey A Douglas,et al.  Higher-order latent trait models for cognitive diagnosis , 2004 .

[10]  Louis Roussos,et al.  THE FUSION MODEL FOR SKILLS DIAGNOSIS: BLENDING THEORY WITH PRACTICALITY , 2008 .

[11]  John T. Willse,et al.  Defining a Family of Cognitive Diagnosis Models Using Log-Linear Models with Latent Variables , 2009 .

[12]  Kikumi K. Tatsuoka,et al.  Architecture of knowledge structures and cognitive diagnosis: A statistical pattern recognition and classification approach. , 1995 .

[13]  B. Junker,et al.  Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory , 2001 .

[14]  Kikumi K. Tatsuoka,et al.  Bug distribution and statistical pattern classification , 1987 .

[15]  K. Tatsuoka.  RULE SPACE: AN APPROACH FOR DEALING WITH MISCONCEPTIONS BASED ON ITEM RESPONSE THEORY , 1983 .