Random Generation of Response Patterns under Computerized Adaptive Testing with the R Package catR

This paper outlines a computerized adaptive testing (CAT) framework and presents an R package for the simulation of response patterns under CAT procedures. This package, called catR, requires a bank of items, previously calibrated according to the four-parameter logistic (4PL) model or any simpler logistic model. The package proposes several methods to select the early test items, several methods for next item selection, different estimators of ability (maximum likelihood, Bayes modal, expected a posteriori, weighted likelihood), and three stopping rules (based on the test length, the precision of ability estimates or the classification of the examinee). After a short description of the different steps of a CAT process, the commands and options of the catR package are presented and practically illustrated.

[1]  Hua-Hua Chang,et al.  The maximum priority index method for severely constrained item selection in computerized adaptive testing. , 2009, The British journal of mathematical and statistical psychology.

[2]  Cornelis A.W. Glas,et al.  Computerized adaptive testing : theory and practice , 2000 .

[3]  Eric Loken,et al.  Estimation of a four-parameter item response theory model. , 2010, The British journal of mathematical and statistical psychology.

[4]  Seung W Choi,et al.  Comparison of CAT Item Selection Criteria for Polytomous Items , 2009, Applied psychological measurement.

[5]  Gilles Raîche,et al.  SIMCAT 1.0: A SAS Computer Program for Simulating Computer Adaptive Testing , 2006 .

[6]  F. Lord.  Applications of Item Response Theory To Practical Testing Problems , 1980 .

[7]  Seung W. Choi,et al.  Firestar: Computerized Adaptive Testing Simulation Program for Polytomous Item Response Theory Models , 2009 .

[8]  Hua-Hua Chang,et al.  Computerized Adaptive Testing: A Comparison of Three Content Balancing Methods , 2003 .

[9]  Kelly L. Rulison,et al.  I've Fallen and I Can't Get Up: Can High-Ability Students Recover From Early Mistakes in CAT? , 2009, Applied psychological measurement.

[10]  Rob R. Meijer,et al.  Computerized Adaptive Testing: Overview and Introduction , 1999 .

[11]  Peter J. Pashley,et al.  Item selection and ability estimation adaptive testing , 2010 .

[12]  D. D. Bickerstaff,et al.  Computerized adaptive testing , 2015 .

[13]  Janice A. Gifford,et al.  Bayesian estimation in the three-parameter logistic model , 1986 .

[14]  Willem J. van der Linden,et al.  Bayesian item selection criteria for adaptive testing , 1998 .

[15]  Melvin R. Novick,et al.  Some latent train models and their use in inferring an examinee's ability , 1966 .

[16]  Allan Birnbaum,et al.  STATISTICAL THEORY FOR LOGISTIC MENTAL TEST MODELS WITH A PRIOR DISTRIBUTION OF ABILITY , 1967 .

[17]  David J. Weiss,et al.  Book Review : New Horizons in Testing: Latent Trait Test Theory and Computerized Adaptive Testing David J. Weiss (Ed.) New York: Academic Press, 1983, 345 pp., $35.00 , 1984 .

[18]  H. Jeffreys,et al.  Theory of probability , 1896 .

[19]  W. J. J. Veerkamp,et al.  Some New Item Selection Criteria for Adaptive Testing , 1994 .

[20]  Anthony R. Zara,et al.  A Comparison of Procedures for Content-Sensitive Item Selection in Computerized Adaptive Tests. , 1991 .

[21]  R. Darrell Bock,et al.  Estimating item parameters and latent ability when responses are scored in two or more nominal categories , 1972 .

[22]  Martha L. Stocking,et al.  Controlling Item Exposure Conditional on Ability in Computerized Adaptive Testing , 1998 .

[23]  E. B. Andersen,et al.  Asymptotic Properties of Conditional Maximum‐Likelihood Estimators , 1970 .

[24]  Cynthia G. Parshall,et al.  Test Development Exposure Control for Adaptive Testing. , 1998 .

[25]  M. Dennis,et al.  A Comparison of Content-Balancing Procedures for Estimating Multiple Clinical Domains in Computerized Adaptive Testing: Relative Precision, Validity, and Detection of Persons With Misfitting Responses , 2010, Applied psychological measurement.

[26]  Frederic M. Lord,et al.  Unbiased estimators of ability parameters, of their variance, and of their parallel-forms reliability , 1983 .

[27]  Cynthia G. Parshall,et al.  New Algorithms for Item Selection and Exposure Control with Computerized Adaptive Testing. , 1995 .

[28]  R. D. Bock,et al.  Marginal maximum likelihood estimation of item parameters , 1982 .

[29]  Cynthia G. Parshall,et al.  Practical Considerations in Computer-Based Testing , 2002 .

[30]  Martha L. Stocking,et al.  A New Method of Controlling Item Exposure in Computerized Adaptive Testing. , 1995 .

[31]  H. Jeffreys.  An invariant form for the prior probability in estimation problems , 1946, Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences.

[32]  Rebecca D. Hetter,et al.  Item exposure control in CAT-ASVAB. , 1997 .

[33]  David Magis,et al.  catR: an R package for computerized adaptive testing , 2011 .

[34]  Bert F. Green.  A Comment on Early Student Blunders on Computer-Based Adaptive Tests , 2011 .

[35]  Anthony R. Zara,et al.  Procedures for Selecting Items for Computerized Adaptive Tests. , 1989 .

[36]  Anastasios A. Economides,et al.  A Review of Item Exposure Control Strategies for Computerized Adaptive Testing Developed from 1983 to 2005 , 2007 .

[37]  J. Mcbride,et al.  Reliability and Validity of Adaptive Ability Tests in a Military Setting , 1983 .

[38]  Frederic M. Lord.  MAXIMUM LIKELIHOOD AND BAYESIAN PARAMETER ESTIMATION IN ITEM RESPONSE THEORY , 1986 .

[39]  Howard Wainer,et al.  Computerized Adaptive Testing: A Primer , 2000 .

[40]  T. A. Warm.  Weighted likelihood estimation of ability in item response theory , 1989 .

[41]  R. D. Bock,et al.  Adaptive EAP Estimation of Ability in a Microcomputer Environment , 1982 .

[42]  Fritz Drasgow,et al.  Innovations in Computerized Assessment , 1999 .

[43]  Peter J. Pashley,et al.  Chapter 1 Item Selection and Ability Estimation in Adaptive Testing , 2000 .

[44]  Erling B. Andersen,et al.  The Numerical Solution of a Set of Conditional Estimation Equations , 1972 .

[45]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[46]  Z. Ying,et al.  a-Stratified Multistage Computerized Adaptive Testing , 1999 .

[47]  Herbert Hoijtink,et al.  On person parameter estimation in the dichotomous Rasch model , 1995 .

[48]  Bor-Yaun Twu,et al.  A Comparative Study of Item Exposure Control Methods in Computerized Adaptive Testing , 1998 .