Content Balancing in Stratified Computerized Adaptive Testing Designs

For computerized adaptive testing (CAT) programs that adopt three-parameter logistic model, high discriminative items generally provide more information for trait estimation. The information based item selection methods in administering computerized adaptive tests (CATs) tend to choose the item that provides maximum information at an examinee’s estimated trait level. As a result, these methods can yield extremely skewed item exposure distribution in which items with high a values may be over-exposed while those with low a values may never be selected. On a different line of thought, Chang and Ying (1999) proposed the a-stratified design (ASTR) that attempts to equalize item exposure distribution by uplifting the usage of low a items. The method has been demonstrated to be effective in improving the utilization of the entire pool without sacrificing the efficiency in ability estimation when it is used with certain types of item pools. Nevertheless, the ASTR may result in a number of items being overexposed in some pools where the correlation between the aand b-parameters is significant. To remedy the over exposure problem, Chang, Qian and Ying (1999) developed the a-stratified with b-blocking method (BASTR) based on ASTR. These two stratified methods have not been tested under situations where content specifications are imposed. For addressing the issue of content balancing, an adaptation of the general ideas of the constrained CAT (CCAT; Kingsbury & Zara, 1989) to ASTR and BASTR was investigated in this study. In addition, the effects of incorporating Sympson-Hetter (SH; Sympson & Hetter, 1985) exposure control into ASTR and BASTR were also examined. The findings indicate that both ASTR and BASTR, with or without SH exposure control, can meet the content specifications, make better utilization of item pools, and yield lower test-overlap rates.

[1]  R. Owen,et al.  A Bayesian Sequential Procedure for Quantal Response in the Context of Adaptive Mental Testing , 1975 .

[2]  Martha L. Stocking,et al.  A New Method of Controlling Item Exposure in Computerized Adaptive Testing. , 1995 .

[3]  Hua-Hua Chang,et al.  A Global Information Approach to Computerized Adaptive Testing , 1996 .

[4]  Frederic M. Lord,et al.  SOME TEST THEORY FOR TAILORED TESTING , 1968 .

[5]  F. Lord.  Applications of Item Response Theory To Practical Testing Problems , 1980 .

[6]  Anthony R. Zara,et al.  Procedures for Selecting Items for Computerized Adaptive Tests. , 1989 .

[7]  Mark D Reckase,et al.  Applied Psychological Measurement , 1977 .

[8]  Linda F. Wightman.  Practical Issues in Computerized Test Assembly , 1998 .

[9]  Martha L. Stocking,et al.  Controlling Item Exposure Conditional on Ability in Computerized Adaptive Testing , 1998 .

[10]  Wayne H. Holtzman,et al.  Computer-assisted instruction, testing, and guidance , 1970 .

[11]  Walter D. Way.  Protecting the Integrity of Computerized Testing Item Pools , 1998 .

[12]  Shu-Ying Chen,et al.  Exploring the Relationship between Item Exposure Rate and Test Overlap Rate in Computerized Adaptive Testing. , 1999 .

[13]  Identifiers California,et al.  Annual Meeting of the National Council on Measurement in Education , 1998 .

[14]  Hua-Hua Chang,et al.  Hypergeometric family and item overlap rates in computerized adaptive testing , 2002 .

[15]  Howard Wainer,et al.  Computerized Adaptive Testing: A Primer , 2000 .

[16]  Martha L. Stocking,et al.  Optimal Design of Item Banks for Computerized Adaptive Tests , 1996 .

[17]  Theo J. H. M. Eggen,et al.  Computerized Adaptive Testing: What It is and How It Works. , 1998 .

[18]  Z. Ying,et al.  a-Stratified Multistage Computerized Adaptive Testing , 1999 .

[19]  Martha L. Stocking,et al.  Practical Issues in Large-Scale Computerized Adaptive Testing , 1996 .

[20]  Wim J. van der Linden,et al.  A Model for Optimal Constrained Adaptive Testing , 1998 .

[21]  David J. Weiss,et al.  Improving Measurement Quality and Efficiency with Adaptive Testing , 1982 .

[22]  Cynthia G. Parshall,et al.  New Algorithms for Item Selection and Exposure Control with Computerized Adaptive Testing. , 1995 .

[23]  Martha L. Stocking,et al.  Controlling Item Exposure Conditional on Ability in Computerized Adaptive Testing , 1995 .

[24]  Hua-Hua Chang,et al.  An Enhanced Stratified Computerized Adaptive Testing Design , 1999 .

[25]  Rebecca D. Hetter,et al.  Item exposure control in CAT-ASVAB. , 1997 .