A Meta-Analysis of Testing Mode Effects in Grade K-12 Mathematics Tests

This study conducted a meta-analysis of computer-based and paper-and-pencil administration mode effects on K-12 student mathematics tests. Both initial and final results based on fixed- and random-effects models are presented. The results based on the final selected studies with homogeneous effect sizes show that the administration mode had no statistically significant effect on K-12 student mathematics tests. Only the moderator variable of computer delivery algorithm contributed to predicting the effect size. The differences in scores between test modes were larger for linear tests than for adaptive tests. However, such variables as study design, grade level, sample size, type of test, computer delivery method, and computer practice did not lead to differences in student mathematics scores between computer-based and paper-and-pencil modes.

[1]  R. Brennan,et al.  Test equating : methods and practices , 1995 .

[2]  G. Glass,et al.  Consequences of Failure to Meet Assumptions Underlying the Fixed Effects Analyses of Variance and Covariance , 1972 .

[3]  Cynthia G. Parshall,et al.  Considerations in Computer-Based Testing , 2002 .

[4]  Michael Russell,et al.  Does it Matter with What I Write? Comparing Performance on Paper, Computer and Portable Writing Devices , 2002 .

[5]  F. Drasgow,et al.  Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis. , 1993 .

[6]  Walt Haney,et al.  Testing Writing on Computers: Results of a Pilot Study To Compare Student Writing Test Performance via Computer or via Paper-and-Pencil , 1996 .

[7]  Richard P. DeShon,et al.  Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. , 2002, Psychological methods.

[8]  W. Dunlap,et al.  Meta-Analysis of Experiments With Matched Groups or Repeated Measures Designs , 1996 .

[9]  G. Gage Kingsbury,et al.  A comparison of achievement level estimates from computerized adaptive testing and paper-and-pencil testing , 1988 .

[10]  G. Neuman,et al.  Computerization of Paper-and-Pencil Tests: When are They Equivalent? , 1998 .

[11]  Kyoko Ito,et al.  Comparability of Scores from Norm-Referenced Paper-and-Pencil and Web-Based Linear Tests for Grades 4 - 12 , 2004 .

[12]  Mark D. Reckase,et al.  TECHNICAL GUIDELINES FOR ASSESSING COMPUTERIZED ADAPTIVE TESTS , 1984 .

[13]  Larry D. Evans,et al.  Children’s reading skills: A comparison of traditional and computerized assessment , 1995 .

[14]  Betty A. Bergstrom,et al.  Ability Measure Equivalence of Computer Adaptive and Pencil and Paper Tests: A Research Synthesis. , 1992 .

[15]  Jong-Pil Kim,et al.  Meta-Analysis of Equivalence of Computerized and P&P Tests on Ability Measures , 1999 .

[16]  Mary Pommerich,et al.  From Simulation to Application: Examinees React to Computerized Testing , 2000 .

[17]  Alija Kulenović,et al.  Standards for Educational and Psychological Testing , 1999 .

[18]  Shudong Wang,et al.  Administration Mode Comparability Study for Stanford Diagnostic Reading and Mathematics Tests , 2004 .

[19]  Daniel R. Eignor,et al.  DERIVING COMPARABLE SCORES FOR COMPUTER ADAPTIVE AND CONVENTIONAL TESTS: AN EXAMPLE USING THE SAT1,2 , 1993 .

[20]  Larry V. Hedges,et al.  How hard is hard science, how soft is soft science? The empirical cumulativeness of research. , 1987 .

[21]  L. Hedges.  Estimation of effect size from a series of independent experiments. , 1982 .

[22]  Cynthia G. Parshall,et al.  Practical Considerations in Computer-Based Testing , 2002 .

[23]  Robert F. Conry,et al.  Effects of Computer-Based Tests on the Achievement, Anxiety, and Attitudes of Grade 10 Science Students , 1991 .

[24]  Ronette L. Kolotkin,et al.  Effects of Computerized Administration on Scores on the Minnesota Multiphasic Personality Inventory , 1977 .

[25]  Identifiers California,et al.  Annual Meeting of the National Council on Measurement in Education , 1998 .

[26]  Betsy Jane Becker,et al.  Synthesizing standardized mean‐change measures , 1988 .

[27]  Daniel J. Mueller,et al.  Implications of Changing Answers on Objective Test Items. , 1977 .

[28]  Edward H. Scissons.  Computer Administration of the California Psychological Inventory. , 1976 .

[29]  L. Hedges.  Distribution Theory for Glass's Estimator of Effect size and Related Estimators , 1981 .

[30]  Walter P. Vispoel.  How Review Options and Administration Modes Influence Scores on Computerized Vocabulary Tests. , 1992 .

[31]  Steven L. Wise,et al.  Research on the Effects of Administering Tests via Computers. , 1989 .

[32]  Mark W. Lipsey,et al.  Practical Meta-Analysis , 2000 .

[33]  Alan C. Bugbee,et al.  The Equivalence of Paper-and-Pencil and Computer-Based Testing. , 1996 .

[34]  Edward M. Levinson,et al.  A Review of the Computerized Version of the Self-Directed Search , 1990 .

[35]  Michael Russell,et al.  Mode of Administration Effects on MCAS Composition Performance for Grades Four, Eight, and Ten. A Report of Findings Submitted to the Massachusetts Department of Education. NBETPP Statements World Wide Web Bulletin. , 2000 .

[36]  Tom Plati,et al.  Effects of Computer Versus Paper Administrations of a State-Mandated Writing Assessment , 2022 .

[37]  Xiao-Hua Zhou,et al.  Statistical Methods for Meta‐Analysis , 2008 .

[38]  Anne L. Harvey,et al.  The Equivalence of Scores from Automated and Conventional Educational and Psychological Tests: A Review of the Literature. College Board Report No. 88-8. , 1988 .

[39]  The International Test Commission.  International Guidelines on Computer-Based and Internet-Delivered Testing , 2006 .

[40]  Michael Russell,et al.  Testing Writing on Computers: An Experiment Comparing Student Performance on Tests Conducted via Computer and via Paper-and-Pencil , 1997 .

[41]  John E. Hunter,et al.  Methods of Meta-Analysis: Correcting Error and Bias in Research Findings , 1991 .