Utilizing Response Time Distributions for Item Selection in CAT

Traditional methods for item selection in computerized adaptive testing only focus on item information without taking into consideration the time required to answer an item. As a result, some examinees may receive a set of items that take a very long time to finish, and information is not accrued as efficiently as possible. The authors propose two item-selection criteria that utilize information from a lognormal model for response times. The first modifies the maximum information criterion to maximize information per time unit. The second is an inverse time-weighted version of a-stratification that takes advantage of the response time model, but achieves more balanced item exposure than the information-based techniques. Simulations are conducted to compare these procedures against their counterparts that ignore response times, and efficiency of estimation, time-required, and item exposure rates are assessed.

[1]  S. Embretson.  A cognitive design system approach to generating valid tests : Application to abstract reasoning , 1998 .

[2]  Daniel O. Segall,et al.  Multidimensional adaptive testing , 1996 .

[3]  Willem J. van der Linden,et al.  Multidimensional Adaptive Testing with a Minimum Error-Variance Criterion , 1999 .

[4]  Hua-Hua Chang,et al.  Kullback–Leibler Information and Its Applications in Multi-Dimensional Adaptive Testing , 2011 .

[5]  Joanna S. Gorin.  Manipulating Processing Difficulty of Reading Comprehension Questions: The Feasibility of Verbal Item Generation , 2005 .

[6]  Z. Ying,et al.  a-Stratified Multistage Computerized Adaptive Testing , 1999 .

[7]  Hua-Hua Chang,et al.  Item Selection in Computerized Adaptive Testing: Should More Discriminating Items be Used First? , 2001 .

[8]  Wim J. van der Linden,et al.  Multidimensional Adaptive Testing with Optimal Design Criteria for Item Selection , 2008, Psychometrika.

[9]  Bradley A. Hanson,et al.  Development and Calibration of an Item Response Model That Incorporates Response Time , 2005 .

[10]  Edward E. Roskam,et al.  Models for Speed and Time-Limit Tests , 1997 .

[11]  David J. Weiss,et al.  Book Review : New Horizons in Testing: Latent Trait Test Theory and Computerized Adaptive Testing David J. Weiss (Ed.) New York: Academic Press, 1983, 345 pp., $35.00 , 1984 .

[12]  Hua-Hua Chang,et al.  A Comparison of Item Selection Rules at the Early Stages of Computerized Adaptive Testing , 2000 .

[13]  Howard Wainer,et al.  Computerized Adaptive Testing: A Primer , 2000 .

[14]  Wim J. van der Linden,et al.  Empirical Initialization of the Trait Estimator in Adaptive Testing , 1999 .

[15]  van der Linden,et al.  A hierarchical framework for modeling speed and accuracy on test items , 2007 .

[16]  Shu-ying Chen,et al.  Effects of Practical Constraints on Item Selection Rules at the Early Stages of Computerized Adaptive Testing , 2004 .

[17]  Hua-Hua Chang,et al.  A Global Information Approach to Computerized Adaptive Testing , 1996 .

[18]  J. Fox,et al.  A Box-Cox normal model for response times. , 2009, The British journal of mathematical and statistical psychology.

[19]  Daniel O. Segall,et al.  General ability measurement: An application of multidimensional item response theory , 2001 .

[20]  Wim J. van der Linden,et al.  Using Response-Time Constraints to Control for Differential Speededness in Computerized Adaptive Testing , 1999 .

[21]  R. Hambleton,et al.  Handbook of Modern Item Response Theory , 1997 .

[22]  Eric Maris,et al.  Additive and multiplicative models for gamma distributed random variables, and their application as psychometric models for response times , 1993 .

[23]  Z. Ying,et al.  a-Stratified Multistage Computerized Adaptive Testing with b Blocking , 2001 .

[24]  Hua-Hua Chang,et al.  The asymptotic posterior normality of the latent trait in an IRT model , 1993 .

[25]  Hartmann Scheiblechner,et al.  Specifically objective stochastic latency mechanisms , 1979 .

[26]  Ricardo Primi,et al.  Complexity of Geometric Inductive Reasoning Tasks: Contribution to the Understanding of Fluid Intelligence. , 2001 .

[27]  Cornelis A.W. Glas,et al.  Statistical Tests of Conditional Independence Between Responses and/or Response Times on Test Items , 2010 .

[28]  David Thissen,et al.  Timed Testing: An Approach Using Item Response Theory , 1983 .