A Strategy for Controlling Item Exposure in Multidimensional Computerized Adaptive Testing

Although computerized adaptive tests have enjoyed tremendous growth, solutions for important problems remain unavailable. One problem is the control of item exposure rate. Because adaptive algorithms are designed to select optimal items, they choose items with high discriminating power. Thus, these items are selected more often than others, leading to both overexposure and underutilization of some parts of the item pool. Overused items are often compromised, creating a security problem that could threaten the validity of a test. Building on a previously proposed stratification scheme to control the exposure rate for one-dimensional tests, the authors extend their method to multidimensional tests. A strategy is proposed based on stratification in accordance with a functional of the vector of the discrimination parameter, which can be implemented with minimal computational overhead. Both theoretical and empirical validation studies are provided. Empirical results indicate significant improvement over the commonly used method of controlling exposure rate that requires only a reasonable sacrifice in efficiency.

[1]  Howard Wainer,et al.  CATs: Whither and whence , 2000 .

[2]  Hua-Hua Chang,et al.  Hypergeometric family and item overlap rates in computerized adaptive testing , 2002 .

[3]  Richard J. Patz,et al.  A Multidimensional Item Response Theory Approach to Simultaneous Ability Estimation. , 2002 .

[4]  Steven J. Fitzpatrick,et al.  A Comparison of Exposure Control Procedures in CAT Systems Based on Different Measurement Models for Testlets , 2003 .

[5]  J. Mcbride,et al.  Reliability and Validity of Adaptive Ability Tests in a Military Setting , 1983 .

[6]  Howard Wainer,et al.  Computerized Adaptive Testing: A Primer , 2000 .

[7]  Z. Ying,et al.  a-Stratified Multistage Computerized Adaptive Testing , 1999 .

[8]  W. Näther Optimum experimental designs , 1994 .

[9]  Frederic M. Lord,et al.  SOME TEST THEORY FOR TAILORED TESTING , 1968 .

[10]  H. Wainer Computerized adaptive testing: A primer, 2nd ed. , 2000 .

[11]  Steven P. Reise,et al.  Computerized Adaptive Testing: A Primer (Second Edition). Howard Wainer (Ed.) (with Neil Dorans, Donald Eignor, Ronald Flaugher, Bert Green, Robert Mislevy, Lynne Steinberg, and David Thissen). [book review] , 2001 .

[12]  Anthony C. Atkinson,et al.  Optimum Experimental Designs , 1992 .

[13]  John Hattie,et al.  Decision Criteria for Determining Unidimensionality , 1981 .

[14]  Howard Wainer Rescuing Computerized Testing by Breaking Zipf's Law , 2000 .

[15]  R. Darrell Bock,et al.  IRT Estimation of Domain Scores , 1997 .

[16]  Cynthia G. Parshall,et al.  Item Exposure Control in Computer-Adaptive Testing: The Use of Freezing to Augment Stratification , 2000 .

[17]  William A. Sands,et al.  Computerized adaptive testing: From inquiry to operation. , 1997 .

[18]  M. Reckase Multidimensional Item Response Theory , 2009 .

[19]  Wim J. van der Linden,et al.  A Model for Optimal Constrained Adaptive Testing , 1998 .

[20]  F. Lord Applications of Item Response Theory To Practical Testing Problems , 1980 .

[21]  Mark D. Reckase,et al.  The Discriminating Power of Items That Measure More Than One Dimension , 1991 .

[22]  Frederic M. Lord ROBBINS‐MONRO PROCEDURES FOR TAILORED TESTING* , 1969 .

[23]  Z. Ying,et al.  a-Stratified Multistage Computerized Adaptive Testing with b Blocking , 2001 .

[24]  Hua-Hua Chang,et al.  Incorporation Of Content Balancing Requirements In Stratification Designs For Computerized Adaptive Testing , 2003 .

[25]  Wen-Chung Wang,et al.  Implementation and Measurement Efficiency of Multidimensional Computerized Adaptive Testing , 2004 .

[26]  Drew Seils,et al.  Optimal design , 2007 .

[27]  Cynthia G. Parshall,et al.  New Algorithms for Item Selection and Exposure Control with Computerized Adaptive Testing. , 1995 .

[28]  Martha L. Stocking,et al.  Controlling Item Exposure Conditional on Ability in Computerized Adaptive Testing , 1995 .

[29]  Mark D. Reckase,et al.  A Linear Logistic Multidimensional Model for Dichotomous Item Response Data , 1997 .

[30]  Daniel O. Segall,et al.  Multidimensional adaptive testing , 1996 .

[31]  Wayne H. Holtzman,et al.  Computer-assisted instruction, testing, and guidance , 1970 .

[32]  P. Laycock,et al.  Optimum Experimental Designs , 1995 .

[33]  J Deborah,et al.  Annual Meeting of the National Council on Measurement in Education , 2000 .

[34]  Timothy R. Miller Empirical Estimation of Standard Errors of Compensatory MIRT Model Parameters Obtained from the NOHARM Estimation Program. ACT Research Report Series. , 1991 .