Using Neural Network Analysis to Define Methods of DINA Model Estimation for Small Sample Sizes

The DINA model is a commonly used model for obtaining diagnostic information. Like many other Diagnostic Classification Models (DCMs), it can require a large sample size to obtain reliable item and examinee parameter estimation. Neural Network (NN) analysis is a classification method that uses a training dataset for calibration. As a result, if this training dataset is determined theoretically, as was the case in Gierl’s attribute hierarchical method (AHM), the NN analysis does not have any sample size requirements. However, a NN approach does not provide traditional item parameters of a DCM or allow for item responses to influence test calibration. In this paper, the NN approach will be implemented for the DINA model estimation to explore its effectiveness as a classification method beyond its use in AHM. The accuracy of the NN approach across different sample sizes, item quality and Q-matrix complexity is described in the DINA model context. Then, a Markov Chain Monte Carlo (MCMC) estimation algorithm and Joint Maximum Likelihood Estimation is used to extend the NN approach so that item parameters associated with the DINA model are obtained while allowing examinee responses to influence the test calibration. The results derived by the NN, the combination of MCMC and NN (NN MCMC) and the combination of JMLE and NN are compared with that of the well-established Hierarchical MCMC procedure and JMLE with a uniform prior on the attribute profile to illustrate their strength and weakness.

[1]  Mark J. Gierl,et al.  Making Diagnostic Inferences about Cognitive Attributes Using the Rule-Space Model and Attribute Hierarchy Method. , 2007 .

[2]  Jeffrey A Douglas,et al.  Higher-order latent trait models for cognitive diagnosis , 2004 .

[3]  J. D. L. Torre,et al.  DINA Model and Parameter Estimation: A Didactic , 2009 .

[4]  Mark J. Gierl,et al.  Using the Attribute Hierarchy Method to Make Diagnostic Inferences about Examinees' Cognitive Skills in Algebra on the SAT. , 2008 .

[5]  J. Templin,et al.  Measurement of psychological disorders using cognitive diagnosis models. , 2006, Psychological methods.

[6]  Richard J. Patz,et al.  A Straightforward Approach to Markov Chain Monte Carlo Methods for Item Response Models , 1999 .

[7]  Jonathan Templin,et al.  Diagnostic Measurement: Theory, Methods, and Applications , 2010 .

[8]  Jimmy de la Torre,et al.  An Empirically Based Method of Q‐Matrix Validation for the DINA Model: Development and Applications , 2008 .

[9]  Mark J. Gierl,et al.  Using Connectionist Models to Evaluate Examinees' Response Patterns on , 2007 .

[10]  Edward H. Haertel.  Using restricted latent class models to map the skill structure of achievement items , 1989 .

[11]  Mark J. Gierl,et al.  The Attribute Hierarchy Method for Cognitive Assessment: A Variation on Tatsuoka's Rule-Space Approach , 2004 .

[12]  G. David Garson,et al.  Neural Networks: An Introductory Guide for Social Scientists , 1999 .

[13]  Mark J. Gierl,et al.  Using the Attribute Hierarchy Method to Identify and Interpret Cognitive Skills that Produce Group Differences. , 2008 .

[14]  B. Junker,et al.  Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory , 2001 .

[15]  Jonathan Templin,et al.  Robustness of Hierarchical Modeling of Skill Association in Cognitive Diagnosis Models , 2008 .

[16]  Ana González Marcos,et al.  AMORE: A MORE flexible neural network package , 2014 .

[17]  Descriptors Evaluators.  Annual Meeting of the National Council on Measurement in Education , 2000 .