Item-Level Comparative Analysis of Online and Paper Administrations of the Texas Assessment of Knowledge and Skills

This article describes a comparative study conducted at the item level for paper and online administrations of a statewide high stakes assessment. The goal was to identify characteristics of items that may have contributed to mode effects. Item-level analyses compared two modes of the Texas Assessment of Knowledge and Skills (TAKS) for up to four subjects at two grade levels. The analyses included significance tests of p-value differences, DIF, and response distributions for each item. Additional analyses investigated item position effects and objective-level mode differences. No evidence of item position effects emerged, but significant differences were found for several items and objectives in all subjects at grade 8 and in mathematics and English language arts (ELA) at grade 11. Differences generally favored the paper group. ELA items that were longer in passage length and math items that required graphing and geometric manipulations or involved scrolling in the online administration tended to be the items showing mode differences.

[1]  Michael Russell,et al.  Testing Writing on Computers: An Experiment Comparing Student Performance on Tests Conducted via Computer and via Paper-and-Pencil , 1997 .

[2]  Brooks Applegate,et al.  Construction of Geometric Analogy Problems by Young Children in a Computer-Based Test , 1993 .

[3]  Georg Rasch,et al.  Probabilistic Models for Some Intelligence and Attainment Tests , 1981, The SAGE Encyclopedia of Research Design.

[4]  Randy Elliot Bennett,et al.  Online Assessment in Mathematics and Writing: Reports from the NAEP Technology-Based Assessment Project, Research and Development Series. NCES 2005-457. , 2005 .

[5]  Walt Haney,et al.  Bridging the Gap between Testing and Technology in Schools , 2000 .

[6]  Michael Russell.  Testing On Computers , 1999 .

[7]  W. Haenszel,et al.  Statistical aspects of the analysis of data from retrospective studies of disease. , 1959, Journal of the National Cancer Institute.

[8]  Craig N. Mills,et al.  FIELD TEST OF A COMPUTER-BASED GRE GENERAL TEST , 1993 .

[9]  Mary Pommerich,et al.  Developing Computerized Versions of Paper-and-Pencil Tests: Mode Effects for Passage-Based Tests , 2004 .

[10]  Andrew J. Poggio,et al.  A Comparative Evaluation of Score Results from Computerized and Paper & Pencil Mathematics Testing in a Large Scale State Assessment Program , 2005 .

[11]  Robert F. Conry,et al.  Effects of Computer-Based Tests on the Achievement, Anxiety, and Attitudes of Grade 10 Science Students , 1991 .

[12]  Steven J. Fitzpatrick,et al.  Score Comparability of Online and Paper Administrations of the Texas Assessment of Knowledge and Skills , 2006 .

[13]  Douglas F. Becker,et al.  The Score Equivalence of Paper-and-Pencil and Computerized Versions of a Speeded Test of Reading Comprehension , 2002 .

[14]  Maria Lee,et al.  Can the Same Results be Obtained Using Computer-mediated Tests as for Paper-based Tests for National Curriculum Assessment? , 2000 .

[15]  Michael Russell,et al.  Testing On Computers , 1999 .