Obtaining Reliable Diagnostic Information through Constrained CAT

Computerized adaptive testing (CAT) has the advantage of delivering tests in an interactive manner such that the ability or latent traits are more effectively estimated. Until now, most research concerning item selection rules in CAT has been built upon either item response theory (IRT) or cognitive diagnostic models (CDM) separately. The only study that combined these two approaches together was done by McGlohen and Chang (2008). They proposed a two-stage method, in which a “shadow” test functioned as a bridge to connect information gathered at  for IRT, and information accumulated at  for CDM. In this paper, we develop a one-stage method to build a CAT featuring reliable cognitive diagnosis. The major idea is to treat diagnostic information as various constraints, and by using a maximum priority index (MPI) method to meet these constraints the cognitive diagnosis can be done reliably at the end of the test. Several priority functions are proposed, some based upon formal measures of information, like Kullback-Leibler information, and others only utilize the knowledge of which items measure what attributes, as provided by the Q matrix. Simulation studies and their results are reported. We show how utilization of information-based methods both yields higher classification rates for cognitive diagnosis and achieve accurate  estimation. Item exposure rates are also considered for all competing methods.