Online Item Calibration for Q-Matrix in CD-CAT

Item replenishment is important for maintaining a large-scale item bank. In this article, the authors consider calibrating new items based on pre-calibrated operational items under the deterministic inputs, noisy-and-gate model, the specification of which includes the so-called Q -matrix, as well as the slipping and guessing parameters. Making use of the maximum likelihood and Bayesian estimators for the latent knowledge states, the authors propose two methods for the calibration. These methods are applicable to both traditional paper–pencil–based tests, for which the selection of operational items is prefixed, and computerized adaptive tests, for which the selection of operational items is sequential and random. Extensive simulations are done to assess and to compare the performance of these approaches. Extensions to other diagnostic classification models are also discussed.

[1]  J. Templin,et al.  Measurement of psychological disorders using cognitive diagnosis models. , 2006, Psychological methods.

[2]  R. Owen,et al.  A Bayesian Sequential Procedure for Quantal Response in the Context of Adaptive Mental Testing , 1975 .

[3]  Hua-Hua Chang,et al.  Online Calibration Methods for the DINA Model with Independent Attributes in CD-CAT , 2012 .

[4]  J. Templin,et al.  Unique Characteristics of Diagnostic Classification Models: A Comprehensive Review of the Current State-of-the-Art , 2008 .

[5]  B. Junker,et al.  Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory , 2001 .

[6]  Sarah M. Hartz,et al.  A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. , 2002 .

[7]  Jae-Chun Ban,et al.  Data Sparseness and On‐Line Pretest Item Calibration‐Scaling Methods in CAT , 2002 .

[8]  Identifiers California,et al.  Annual Meeting of the National Council on Measurement in Education , 1998 .

[9]  Howard Wainer,et al.  Computerized Adaptive Testing: A Primer , 2000 .

[10]  Jonathan Templin,et al.  Diagnostic Measurement: Theory, Methods, and Applications , 2010 .

[11]  Jingchen Liu,et al.  A Rate Function Approach to Computerized Adaptive Testing for Cognitive Diagnosis , 2015, Psychometrika.

[12]  Chia-Yi Chiu,et al.  Cluster Analysis for Cognitive Diagnosis: Theory and Applications , 2009 .

[13]  Martha L. Stocking.  Scale Drift in On-Line Calibration. , 1988 .

[14]  Matthias von Davier,et al.  A GENERAL DIAGNOSTIC MODEL APPLIED TO LANGUAGE TESTING DATA , 2005 .

[15]  K. Tatsuoka.  RULE SPACE: AN APPROACH FOR DEALING WITH MISCONCEPTIONS BASED ON ITEM RESPONSE THEORY , 1983 .

[16]  Curtis Tatsuoka,et al.  Data analytic methods for latent partially ordered classification models , 2002 .

[17]  Chen Ping,et al.  Item Replenishing in Cognitive Diagnostic Computerized Adaptive Testing , 2011 .

[18]  Hua-Hua Chang,et al.  A Simulation Study to Compare CAT Strategies for Cognitive Diagnosis , 2003 .

[19]  Cornelis A.W. Glas,et al.  Computerized adaptive testing : theory and practice , 2000 .

[20]  Neil J. Dorans,et al.  Item Response Theory, Item Calibration, and Proficiency Estimation , 2000 .

[21]  K. Tatsuoka.  Cognitive Assessment: An Introduction to the Rule Space Method , 2009 .

[22]  Willem J. van der Linden,et al.  Bayesian item selection criteria for adaptive testing , 1998 .

[23]  Edward H. Haertel.  Using restricted latent class models to map the skill structure of achievement items , 1989 .

[24]  F. Lord.  Applications of Item Response Theory To Practical Testing Problems , 1980 .

[25]  Ying Cheng.  When Cognitive Diagnosis Meets Computerized Adaptive Testing: CD-CAT , 2009 .

[26]  Frederic M. Lord.  ROBBINS‐MONRO PROCEDURES FOR TAILORED TESTING* , 1969 .

[27]  Daniel O. Segall.  Calibrating CAT Pools and Online Pretest Items Using MCMC Methods , 2003 .

[28]  Mark J. Gierl,et al.  The Attribute Hierarchy Method for Cognitive Assessment: A Variation on Tatsuoka's Rule-Space Approach , 2004 .

[29]  J. Templin,et al.  The Effects of Q-Matrix Misspecification on Parameter Estimates and Classification Accuracy in the DINA Model , 2008 .

[30]  Jae-Chun Ban,et al.  A Comparative Study of On‐line Pretest Item—Calibration/Scaling Methods in Computerized Adaptive Testing , 2001 .

[31]  John T. Willse,et al.  Defining a Family of Cognitive Diagnosis Models Using Log-Linear Models with Latent Variables , 2009 .