Optimal item pool design for a highly constrained computerized adaptive test

Item pool quality has been regarded as one important factor to help realize enhanced measurement quality for the computerized adaptive test (CAT) (e.g., Flaugher, 2000; Jensema, 1977; McBride & Wise, 1976; Reckase, 1976; 2003; van der Linden, Ariel, & Veldkamp, 2006; Veldkamp & van der Linden, 2000; Xing & Hambleton, 2004). However, studies are rare in how to identify the desired features of an item pool for the computerized adaptive test (CAT). Unlike the problem of item pool assembly in which an item pool is assembled from an available master pool according to the desired specifications, no actual items are available yet in the problem of item pool design (van der Linden, Ariel, & Veldkamp, 2006). Since there is no actual item available when designing an item pool, designing an item pool that is optimal intuitively becomes a desired goal. This study is focused on designing an optimal item pool for a CAT using the weighted deviations model (WDM; Stocking & Swanson, 1993) item selection procedure. Drawing on Reckase (2003) and Gu (2007), this study has extended the binand-union method proposed by Reckase (2003) to a CAT with a large set of complex non-statistical constraints. The method used to generate optimal item features is a combination of methods based on McBride & Weiss (1976) and Gu (2007) for statistical features and a sampling method based on test specifications for non-statistical features. The end-product is an item blueprint describing items' statistical and non-statistical attributes, item number distribution, and optimal item pool size. A large-scale operational CAT program served as the CAT template in this study. Three key factors considered to potentially impact optimal item pool features were manipulated including item generation method, expected amount of item information change, and b-bin width. Optimal item pool performance was evaluated and compared with that of an operational item pool in light of a series of criteria including measurement accuracy and precision, item pool utilization, test security, constraint violation, and classification accuracy. A demonstrative example on how to use identified optimal item pool features for item pool assembly was provided. How to apply optimal item pool features to item pool management, operational item pool assembly, and item writing was also discussed.

[1]  Steven Buyske,et al.  Optimal Design in Educational Testing , 2005 .

[2]  Carl J. Jensema Bayesian Tailored Testing and the Influence of Item Bank Characteristics , 1977 .

[3]  Walter D. Way Protecting the Integrity of Computerized Testing Item Pools , 1998 .

[4]  van der Linden,et al.  Constrained adaptive testing with shadow tests , 2000 .

[5]  Wim J. van der Linden,et al.  A Model for Optimal Constrained Adaptive Testing , 1998 .

[6]  Wim J. van der Linden,et al.  Capitalization on Item Calibration Error in Adaptive Testing , 1998 .

[7]  Willem J. van der Linden A Comparison of Item-Selection Methods for Adaptive Tests with Content Constraints , 2005 .

[8]  Shu-ying Chen,et al.  Effects of Practical Constraints on Item Selection Rules at the Early Stages of Computerized Adaptive Testing , 2004 .

[9]  Bernard P. Veldkamp,et al.  Designing item pools for computerized adaptive testing , 1999 .

[10]  Steven J. Fitzpatrick,et al.  Computerized Adaptive Testing With the Partial Credit Model: Estimation Procedures, Population Distributions, and Item Pool Characteristics , 2005 .

[11]  Hua-Hua Chang,et al.  The maximum priority index method for severely constrained item selection in computerized adaptive testing. , 2009, The British journal of mathematical and statistical psychology.

[12]  Vern W. Urry,et al.  TAILORED TESTING: A SUCCESSFUL APPLICATION OF LATENT TRAIT THEORY* , 1977 .

[13]  Barbara G. Dodd,et al.  Computerized Adaptive Testing With Polytomous Items , 1995 .

[14]  Ronald K. Hambleton,et al.  Impact of Test Design, Item Quality, and Item Bank Size on the Psychometric Properties of Computer-Based Credentialing Examinations , 2004 .

[15]  Judith A. Spray,et al.  The Relationship Between Item Exposure and Test Overlap in Computerized Adaptive Testing , 2003 .

[16]  Bor-Yaun Twu,et al.  A Comparative Study of Item Exposure Control Methods in Computerized Adaptive Testing , 1998 .

[17]  F. Lord Applications of Item Response Theory To Practical Testing Problems , 1980 .

[18]  Bernard P. Veldkamp,et al.  Constructing rotating item pools for constrained adaptive testing , 2002 .

[19]  G. Gage Kingsbury,et al.  Practical issues in developing and maintaining acomputerized adaptive testing program , 2000 .

[20]  Melvin R. Novick,et al.  Some latent train models and their use in inferring an examinee's ability , 1966 .

[21]  Martha L. Stocking,et al.  A Method for Severely Constrained Item Selection in Adaptive Testing , 1992 .

[22]  Z. Ying,et al.  a-Stratified Multistage Computerized Adaptive Testing , 1999 .

[23]  Willem J. van der Linden,et al.  Optimal Assembly of Psychological and Educational Tests , 1998 .