Mutual Information Item Selection Method in Cognitive Diagnostic Computerized Adaptive Testing With Short Test Length

Cognitive diagnostic computerized adaptive testing (CD-CAT) purports to combine the strengths of both CAT and cognitive diagnosis. Cognitive diagnosis models aim at classifying examinees into the correct mastery profile group so as to pinpoint the strengths and weakness of each examinee whereas CAT algorithms choose items to determine those strengths and weakness as efficiently as possible. Most of the existing CD-CAT item selection algorithms are evaluated when test length is relatively long whereas several applications of CD-CAT, such as in interim assessment, require an item selection algorithm that is able to accurately recover examinees’ mastery profile with short test length. In this article, we introduce the mutual information item selection method in the context of CD-CAT and then provide a computationally easier formula to make the method more amenable in real time. Mutual information is then evaluated against common item selection methods, such as Kullback–Leibler information, posterior weighted Kullback–Leibler information, and Shannon entropy. Based on our simulations, mutual information consistently results in nearly the highest attribute and pattern recovery rate in more than half of the conditions. We conclude by discussing how the number of attributes, Q-matrix structure, correlations among the attributes, and item quality affect estimation accuracy.

[1]  Jeffrey A Douglas,et al.  Higher-order latent trait models for cognitive diagnosis , 2004 .

[2]  Alexander Weissman,et al.  Mutual Information Item Selection in Adaptive Classification Testing , 2007 .

[3]  Curtis Tatsuoka,et al.  Data analytic methods for latent partially ordered classification models , 2002 .

[4]  Wen-Chung Wang,et al.  Improving measurement precision of test batteries using multidimensional item response models. , 2004, Psychological methods.

[5]  David J. Weiss,et al.  Improving Measurement Quality and Efficiency with Adaptive Testing , 1982 .

[6]  B. Junker,et al.  Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory , 2001 .

[7]  Richard J. Patz,et al.  Making the Most of What We Have: A Practical Application of Multidimensional Item Response Theory in Test Scoring , 2005 .

[8]  Hua-Hua Chang,et al.  A Comparison of Item Selection Rules at the Early Stages of Computerized Adaptive Testing , 2000 .

[9]  Hua-Hua Chang,et al.  Restrictive Stochastic Item Selection Methods in Cognitive Diagnostic Computerized Adaptive Testing. , 2011 .

[10]  Hua-Hua Chang,et al.  The maximum priority index method for severely constrained item selection in computerized adaptive testing. , 2009, The British journal of mathematical and statistical psychology.

[11]  Bernard P. Veldkamp,et al.  Multidimensional adaptive testing with constraints on test content , 2002 .

[12]  Russell G. Almond,et al.  Bayes Nets in Educational Assessment: Where Do the Numbers Come from? CSE Technical Report. , 2000 .

[13]  Theodorus Johannes Hendrikus Maria Eggen,et al.  Item Selection in Adaptive Testing with the Sequential Probability Ratio Test , 1999 .

[14]  Ding Shuliang,et al.  Test Construction for Cognitive Diagnosis , 2010, 2010 Asia-Pacific Conference on Wearable Computing Systems.

[15]  Jingchen Liu,et al.  Data-Driven Learning of Q-Matrix , 2012, Applied psychological measurement.

[16]  Kentaro Yamamoto,et al.  HYBRID MODEL OF IRT AND LATENT CLASS MODELS , 1982 .

[17]  Thomas S. Ferguson,et al.  Sequential classification on partially ordered sets , 2003 .

[18]  Ying Cheng.  When Cognitive Diagnosis Meets Computerized Adaptive Testing: CD-CAT , 2009 .

[19]  J. Templin,et al.  Skills Diagnosis Using IRT-Based Latent Class Models , 2007 .

[20]  K. Tatsuoka.  RULE SPACE: AN APPROACH FOR DEALING WITH MISCONCEPTIONS BASED ON ITEM RESPONSE THEORY , 1983 .

[21]  Wim J. van der Linden,et al.  Multidimensional adaptive testing with Kullback-Leibler information item selection , 2009 .

[22]  Edward H. Haertel.  Using restricted latent class models to map the skill structure of achievement items , 1989 .

[23]  Russell G. Almond,et al.  Bayes Nets in Educational Assessment: Where the Numbers Come From , 1999, UAI.

[24]  L. T. DeCarlo.  On the Analysis of Fraction Subtraction Data: The DINA Model, Classification, Latent Class Sizes, and the Q-Matrix , 2011 .

[25]  Jonathan Templin,et al.  Diagnostic Measurement: Theory, Methods, and Applications , 2010 .

[26]  Hua-Hua Chang,et al.  Item Selection in Multidimensional Computerized Adaptive Testing—Gaining Information from Different Angles , 2011 .

[27]  Hua-Hua Chang,et al.  Combining CAT with cognitive diagnosis: A weighted item selection approach , 2012, Behavior research methods.

[28]  Louis V. DiBello,et al.  Guest Editors' Introduction and Overview: IRT‐Based Cognitive Diagnostic Models and Related Methods , 2007 .

[29]  Shameem Nyla.  NATIONAL COUNCIL ON MEASUREMENT IN EDUCATION , 2004 .

[30]  Alan Huebner,et al.  An Overview of Recent Developments in Cognitive Diagnostic Computer Adaptive Assessments. , 2010 .

[31]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[32]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[33]  Hua-Hua Chang,et al.  The Development of Computerized Adaptive Testing with Cognitive Diagnosis for an English Achievement Test in China , 2013, J. Classif..

[34]  M. Reckase.  Multidimensional Item Response Theory , 2009 .

[35]  Mark J. Gierl,et al.  Defining and Evaluating Models of Cognition Used in Educational Measurement to Make Inferences about Examinees' Thinking Processes. , 2007 .

[36]  Jimmy de la Torre,et al.  An Empirically Based Method of Q‐Matrix Validation for the DINA Model: Development and Applications , 2008 .

[37]  Hua-Hua Chang,et al.  Online Calibration Methods for the DINA Model with Independent Attributes in CD-CAT , 2012 .

[38]  J. Templin,et al.  Measurement of psychological disorders using cognitive diagnosis models. , 2006, Psychological methods.

[39]  E. Maris.  Estimating multiple classification latent class models , 1999 .

[40]  Kikumi K. Tatsuoka,et al.  Architecture of knowledge structures and cognitive diagnosis: A statistical pattern recognition and classification approach. , 1995 .

[41]  Tao Xin,et al.  The Application of the Monte Carlo Approach to Cognitive Diagnostic Computerized Adaptive Testing With Content Constraints , 2013 .

[42]  Hua-Hua Chang,et al.  A Global Information Approach to Computerized Adaptive Testing , 1996 .

[43]  Mark J. Gierl,et al.  Estimating Classification Consistency and Accuracy for Cognitive Diagnostic Assessment. , 2012 .