Comparability of Computerized Adaptive and Paper-Pencil Tests

When a traditional Paper-Pencil Test (PPT) is delivered by computer, two types of computerization can be implemented. One is a linear Computer-Based Test (CBT) in which the paper version of the test is presented and administered via computers. In a linear CBT, the items on both versions are identical, in general, and scoring methods and procedures are the same. The change from PPT to CBT, therefore, only involves the change of administration mode. The other type of computerization is the Computerized Adaptive Testing (CAT) in which not only the medium of administration changes from paper to computer but also the test delivery algorithm turns from linear to adaptive. This adaptive testing paradigm allows the test items to be selected and administered so that they are tailored to each test taker’s ability. Therefore, in comparability studies, both the administration mode and paradigm effect on examinees’ performance should be examined to ensure the comparability of the CAT and its PPT counterpart. The administration mode and paradigm effect on examinees’ performance should be examined to ensure the comparability of the CAT and its PPT counterpart. Paradigm Effects The administration mode effect has been widely examined in the comparison of PPT and linear CBT. Although findings are not conclusive, there seems to be a trend indicating that the two versions are comparable across the administration mode (e.g. Paek, 2005, Wang, Jiao, Young, Brooks, & Olson 2007, 2008). When CAT is compared to its PPT counterpart, the mode effect and paradigm effect are confounded with each other. In order to separate the two effects and examine the paradigm effect, some studies have focused on comparability analysis between the linear CBT and CAT.

[1]  G. Neuman,et al.  Computerization of Paper-and-Pencil Tests: When are They Equivalent? , 1998 .

[2]  Mark D. Reckase,et al.  TECHNICAL GUIDELINES FOR ASSESSING COMPUTERIZED ADAPTIVE TESTS , 1984 .

[3]  M. Lunz,et al.  Equating Computerized Adaptive Certification Examinations: The Board of Registry Series of Studies. , 1995 .

[4]  Gary L. Thomasson.  The Goal of Equity within and between Computerized Adaptive Tests and Paper and Pencil Forms. , 1997 .

[5]  Mary Pommerich,et al.  The Effect of Using Item Parameters Calibrated from Paper Administrations in Computer Adaptive Test Administrations , 2007 .

[6]  Hong Jiao,et al.  Comparability of Computer-Based and Paper-and-Pencil Testing in K–12 Reading Assessments , 2008 .

[7]  Tianyou Wang,et al.  Evaluating Comparability in Computerized Adaptive Testing: Issues, Criteria and an Example , 2001 .

[8]  Daniel R. Eignor,et al.  DERIVING COMPARABLE SCORES FOR COMPUTER ADAPTIVE AND CONVENTIONAL TESTS: AN EXAMPLE USING THE SAT1,2 , 1993 .

[9]  Shudong Wang,et al.  A Meta-Analysis of Testing Mode Effects in Grade K-12 Mathematics Tests , 2007 .

[10]  Daniel O. Segall,et al.  Equating the CAT-ASVAB. , 1997 .

[11]  Walter P. Vispoel,et al.  Individual Differences and Test Administration Procedures: A Comparison of Fixed-Item, Computerized-Adaptive, and Self-Adapted Testing. , 1994 .

[12]  Martha L. Stocking,et al.  A Method for Severely Constrained Item Selection in Adaptive Testing , 1992 .

[13]  M. J. Kolen.  Threats to Score Comparability with Applications to Performance Assessments and Computerized Adaptive Tests , 1999 .

[14]  Douglas F. Becker,et al.  The Score Equivalence of Paper-and-Pencil and Computerized Versions of a Speeded Test of Reading Comprehension , 2002 .

[15]  F. Drasgow,et al.  Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis. , 1993 .

[16]  Tianyou Wang,et al.  Computerized Adaptive and Fixed‐Item Testing of Music Listening Skill: A Comparison of Efficiency, Precision, and Concurrent Validity , 1997 .

[17]  Tony Thompson,et al.  Investigating CAT Designs to Achieve Comparability With a Paper Test , 2007 .

[18]  Gary A. Schaeffer.  The Introduction and Comparability of the Computer Adaptive GRE General Test. GRE Board Professional Report No. 88-08aP. , 1995 .