Replacing Paper-Based Testing with an Alternative for the Assessment of Iranian Undergraduate Students: Administration Mode Effect on Testing Performance

There have been studies on comparability of test results in Computer-Based Testing (Henceforth CBT) and Paper-Based Testing (Henceforth PBT) considering key factors associated with test results in different countries with different languages and technological backgrounds. The main purpose of the current study was to discover the equivalency of test scores on PBT and CBT in the English achievement test in Payame Noor University (PNU) among undergraduate students. It also intended to investigate if there was any relationship between computer attitude and testing performance on CBT. Based upon the quantitative and qualitative data, some major findings were revealed. Firstly, there was statistically significant difference between two sets of mean scores. Furthermore, based on descriptive results, in comparing the results of computerized and paper-based tests, students showed better performance on PBT than CBT. The results of this study support the necessity of doing comparability studies in higher educational contexts before substituting CBT for PBT or including it in the system. Then, computer attitude had not any interaction with testing performance on CBT among Iranian undergraduate students in PNU. Finally, the results of interview supported the quantitative findings, i.e. participants mostly showed high preference for computerized test and liked CBT more than PBT but due to some justifications and habit of taking tests traditionally, they performed better on PBT.

[1]  Sara A. Brallier,et al.  Computer-Based Testing: A Comparison of Computer-Based and Paper-and-Pencil Assessment , 2010 .

[2]  Judith O'Donnell Dooling.  What Students Want To Learn about Computers , 2000 .

[3]  J. B. Olsen,et al.  THE FOUR GENERATIONS OF COMPUTERIZED EDUCATIONAL MEASUREMENT , 1988 .

[4]  R. Warner.  Applied Statistics: From Bivariate through Multivariate Techniques [with CD-ROM]. , 2007 .

[5]  Walter P. Vispoel.  Reviewing and Changing Answers on Computer‐adaptive and Self‐adaptive Vocabulary Tests , 1998 .

[6]  Michael Levy,et al.  Computer applications in second language acquisition : Foundations for teaching , testing and research , 2009 .

[7]  Robert F. McMorris,et al.  Reasons for Changing Answers: An Evaluation Using Personal Interviews. , 1991 .

[8]  Randy Elliot Bennett,et al.  Does It Matter if I Take My Mathematics Test on Computer? A Second Empirical Study of Mode Effects in NAEP. , 2008 .

[9]  Performance Differences According to Test Mode and Computer Familiarity on a Practice GRE , 2002 .

[10]  Timothy D. Ritchie,et al.  Factors in Paper-and-Pencil and Computer Reading Score Differences at the Primary Grades , 2006 .

[11]  Jinan Hatem Issa,et al.  TESOL in-Service Teachers’ Attitudes towards Computer Use , 2011 .

[12]  Roy B. Clariana,et al.  Paper-based versus computer-based assessment: key factors associated with the test mode effect , 2002, Br. J. Educ. Technol..

[13]  Gautam Puhan,et al.  Examining Differences in Examinee Performance in Paper and Pencil and Computerized Testing , 2007 .

[14]  Cynthia G. Parshall,et al.  Computer Testing versus Paper-and-Pencil Testing: An Analysis of Examinee Characteristics Associated with Mode Effect. , 1993 .

[15]  Randy Elliot Bennett,et al.  Inexorable and Inevitable: The Continuing Story of Technology and Assessment , 2002 .

[16]  F. Drasgow,et al.  Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis. , 1993 .

[17]  Donald E. Powers,et al.  Inexperienced and Anxious Computer Users: Coping With a Computer-Administered Test of Academic Skills , 1993 .

[18]  Preston Lewis,et al.  A Comparison of Computer-Based Testing and Pencil-and-Paper Testing for Students with a Read-Aloud Accommodation , 2011 .

[19]  Brandon Bennett,et al.  Reinventing Assessment: Speculations on the Future of Large Scale Educational Testing , 1998 .

[20]  H. Leeson.  The Mode Effect: A Literature Review of Human and Technological Issues in Computerized Testing , 2006 .

[21]  I. Seidman.  Interviewing as qualitative research : a guide for researchersin education and the social sciences , 1991 .

[22]  Reima Abobaker,et al.  Assessing Language through Computer Technology , 2012 .

[23]  Lovemore J. Nyaumwe,et al.  Computerized Summative Assessment of Multiple-choice Questions: Exploring Possibilities with the Zimbabwe School Examination Council Grade 7 Assessments , 2011, Comput. Inf. Sci..

[24]  Kenneth Tait,et al.  Computer or paper? That is the question: does the medium in which assessment questions are presented affect children's performance in mathematics? , 2004 .

[25]  Mary Pommerich,et al.  Developing Computerized Versions of Paper-and-Pencil Tests: Mode Effects for Passage-Based Tests , 2004 .

[26]  Anastasios A. Economides,et al.  Computer based assessment: Gender differences in perceptions and acceptance , 2011, Comput. Hum. Behav..

[27]  北市 陽一,et al.  Language Testing , 1964, Language Teaching.

[28]  Thomas Hoffmann,et al.  Examining the Effect of Computer-Based Passage Presentation on Reading Test Performance , 2005 .

[29]  Bridget Anakwe.  Comparison of Student Performance in Paper-Based Versus Computer-Based Testing , 2008 .

[30]  Justin Kruger,et al.  Counterfactual thinking and the first instinct fallacy. , 2005, Journal of Personality and Social Psychology.

[31]  Irwin S. Kirsch,et al.  Examining the Relationship Between Computer Familiarity and Performance on Computer‐Based Language Tasks , 1999 .

[32]  Yasuyo Sawaki,et al.  Comparability of Conventional and Computerized Tests of Reading in a Second Language , 2001 .

[33]  Steven M. Smith,et al.  Student Choice between Computer and Traditional Paper-and-Pencil University Tests: What Predicts Preference and Performance? , 2009 .

[34]  Identifiers California,et al.  Annual Meeting of the National Council on Measurement in Education , 1998 .

[35]  Seyyed Morteza Hashemi Toroujeni,et al.  Cross-Mode Comparability of Computer-Based Testing (CBT) versus Paper-Pencil Based Testing (PPT): An Investigation of Testing Administration Mode among Iranian Intermediate EFL Learners. , 2017 .

[36]  P. Pamela,et al.  Recent Trends in Comparability Studies , 2005 .

[37]  Muhammad Affan Ramadhana.  The Technology and Second Language Acquisition , 2013 .

[38]  Mary Peat,et al.  Supporting student learning: the use of computer-based formative assessment modules , 2002, Br. J. Educ. Technol..

[39]  Sylvia Green,et al.  On-line Mathematics Assessment: The Impact of Mode on Performance and Question Answering Strategies , 2006 .

[40]  Lyle F. Bachman.  Modern language testing at the turn of the century: assuring that what we count counts , 2000 .

[41]  J. Evans.  Straightforward Statistics for the Behavioral Sciences , 1995 .

[42]  Jaeyool Boo,et al.  Comparability of a paper-based language test and a computer-based language test , 2003 .

[43]  G. Neuman,et al.  Computerization of Paper-and-Pencil Tests: When are They Equivalent? , 1998 .

[44]  Steven L. Wise,et al.  Computer-Based Testing in Higher Education. , 1990 .

[45]  Daniel J. Bernstein,et al.  An Examination of the Equivalence between Non-Adaptive Computer-Based and Traditional Testing , 2001 .

[46]  Mark Pomplun,et al.  The Score Comparability of Computerized and Paper-and-Pencil Formats for K-3 Reading Tests , 2005 .

[47]  Seyyed Morteza Hashemi Toroujeni,et al.  Comparability of Computer-Based Testing and Paper-Based Testing: Testing Mode Effect, Testing Mode Order, Computer Attitudes and Testing Mode preference , 2017 .

[48]  David E Vance,et al.  The relationship between computer experience and computerized cognitive test performance among older adults. , 2013, The journals of gerontology. Series B, Psychological sciences and social sciences.

[49]  Dan Douglas,et al.  Assessing Languages for Specific Purposes , 1999 .

[50]  Glenn Fulcher,et al.  Computerizing an English language placement test , 1999 .

[51]  Ghada K. Eid.  An Investigation into the Effects and Factors Influencing Computer-Based Online Math Problem-Solving in Primary Schools , 2005 .