Automated Test Assembly for Cognitive Diagnosis Models Using a Genetic Algorithm

Much recent psychometric literature has focused on cognitive diagnosis models (CDMs), a promising class of instruments used to measure the strengths and weaknesses of examinees. This article introduces a genetic algorithm to perform automated test assembly alongside CDMs. The algorithm is flexible in that it can be applied whether the goal is to minimize the average number of classification errors, minimize the maximum error rate across all attributes being measured, hit a target set of error rates, or optimize any other prescribed objective function. Under multiple simulation conditions, the algorithm compared favorably with a standard method of automated test assembly, successfully finding solutions that were appropriate for each stated goal.

[1]  Louis A. Roussos,et al.  The fusion model skills diagnosis system , 2007 .

[2]  R. Henson,et al.  Cognitive Diagnostic Attribute-Level Discrimination Indices , 2008 .

[3]  Willem J. van der Linden,et al.  Linear Models for Optimal Test Design , 2005 .

[4]  Matthias von Davier,et al.  A General Diagnostic Model Applied to Language Testing Data. Research Report. ETS RR-05-16. , 2005 .

[5]  Martha L. Stocking,et al.  Controlling Item Exposure Conditional on Ability in Computerized Adaptive Testing , 1998 .

[6]  John H. Holland,et al.  Genetic Algorithms and the Optimal Allocation of Trials , 1973, SIAM J. Comput..

[7]  Melvin R. Novick,et al.  Some latent train models and their use in inferring an examinee's ability , 1966 .

[8]  W. D. Linden,et al.  A maximin model for IRT-based test design with practical constraints , 1989 .

[9]  Kikumi K. Tatsuoka,et al.  A Probabilistic Model for Diagnosing Misconceptions By The Pattern Classification Approach , 1985 .

[10]  Jeffrey A Douglas,et al.  Higher-order latent trait models for cognitive diagnosis , 2004 .

[11]  Sandip Sinharay,et al.  Model Diagnostics for Bayesian Networks , 2004 .

[12]  F. Lord.  Applications of Item Response Theory To Practical Testing Problems , 1980 .

[13]  Edward Haertel,et al.  An Application of Latent Class Models to Assessment Data , 1984 .

[14]  Eunice Eunhee Jang,et al.  A *Validity Narrative: Effects of Reading Skills Diagnosis on Teaching and Learning in the Context of NG TOEFL , 2005 .

[15]  B. Veldkamp.  Multiple Objective Test Assembly Problems , 1998 .

[16]  S. Chipman,et al.  Cognitively diagnostic assessment , 1995 .

[17]  Angela J. Verschoor,et al.  Genetic Algorithms for Automated Test Assembly , 2007 .

[18]  T. Theunissen.  Binary programming and test design , 1985 .

[19]  Hua-Hua Chang,et al.  A Global Information Approach to Computerized Adaptive Testing , 1996 .

[20]  Mark D. Reckase,et al.  The Difficulty of Test Items That Measure More Than One Ability , 1985 .

[21]  J. Templin,et al.  Skills Diagnosis Using IRT-Based Latent Class Models , 2007 .

[22]  Jeffrey A Douglas,et al.  Test Construction for Cognitive Diagnosis , 2005 .

[23]  Koun-Tem Sun,et al.  Creating IRT-Based Parallel Test Forms Using the Genetic Algorithm Method , 2008 .

[24]  E. Maris.  Estimating multiple classification latent class models , 1999 .

[25]  B. Junker,et al.  Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory , 2001 .

[26]  William Stout,et al.  The theoretical detect index of dimensionality and its application to approximate simple structure , 1999 .