Comparability of a paper-based language test and a computer-based language test

With the advent of the digital revolution, language testers have endeavored to utilize state-of-the-art computer technology to satisfy the ever-growing need for a tool to measure English communication skills with maximal accuracy and efficiency. Thanks to the concerted efforts made by experts in such fields as computational linguistics, computer engineering, computer-assisted language learning, and psychometrics, language testers have recently succeeded in developing computer/web-based language tests. Among them are the TOEFL CBT by Educational Testing Service and CommuniCAT by the University of Cambridge Local Examinations Syndicate. As with the paper-based language test (PBLT), more rigorous research is now being conducted on the validity of computer-based language tests (CBLT) and computer adaptive language tests (CALT). Content analyses and comparability studies of PBLT and CBLT/CALT are prerequisites to such validation research. In this context, utilizing an EFL test battery entitled the Test of English Proficiency developed by Seoul National University (TEPS), the present study is aimed at addressing the issue of the comparability between PBLT and CBLT based on content and construct validation employing content analyses based on corpus linguistic techniques in addition to such statistical analyses as correlational analyses, ANOVA, and confirmatory factor analyses. The findings support comparability between the CBLT version and the PBLT version of the TEPS subtests (listening comprehension, grammar, vocabulary, and reading comprehension) in question.

[1]  Anne L. Harvey,et al.  The Equivalence of Scores from Automated and Conventional Educational and Psychological Tests: A Review of the Literature. College Board Report No. 88-8. , 1988 .

[2]  J. S. Long,et al.  Testing Structural Equation Models , 1993 .

[3]  Micheline Chalhoub-Deville,et al.  Issues in Computer-Adaptive Testing of Reading Proficiency , 2000 .

[4]  Joan Jamieson,et al.  THE RELATIONSHIP BETWEEN COMPUTER FAMILIARITY AND PERFORMANCE ON COMPUTER-BASED TOEFL TEST TASKS , 1998 .

[5]  Identifiers California,et al.  Annual Meeting of the National Council on Measurement in Education , 1998 .

[6]  Karl G. Jöreskog,et al.  Lisrel 8: User's Reference Guide , 1997 .

[7]  G. Neuman,et al.  Computerization of Paper-and-Pencil Tests: When are They Equivalent? , 1998 .

[8]  Tianyou Wang,et al.  Evaluating Comparability in Computerized Adaptive Testing: Issues, Criteria and an Example , 2001 .

[9]  Dorry M. Kenyon,et al.  COMPARING EXAMINEE ATTITUDES TOWARD COMPUTER- ASSISTED AND OTHER ORAL PROFICIENCY ASSESSMENTS , 2001 .

[10]  M Burgess,et al.  Future challenges. , 1982, The New Zealand nursing journal. Kai tiaki.

[11]  C. Chapelle.  Construct definition and validity inquiry in SLA research , 1999 .

[12]  Michael Russell,et al.  Testing Writing on Computers: An Experiment Comparing Student Performance on Tests Conducted via Computer and via Paper-and-Pencil , 1997 .

[13]  Lyle F. Bachman.  语言测试要略 = Fundamental considerations in language testing , 1990 .

[14]  Andrew D. Cohen,et al.  Interfaces between Second Language Acquisition and Language Testing Research: List of contributors , 1999 .

[15]  Lyle F. Bachman,et al.  An investigation into the adequacy of three IRT models for data from two EFL reading tests , 1992 .

[16]  Test Fairness and Validity of the TEPS , 1999 .

[17]  H. Marsh,et al.  A Multidimensional Physical Self-concept and Its Relations to Multiple Components of Physical Fitness , 1994 .

[18]  Yasuyo Sawaki,et al.  Comparability of Conventional and Computerized Tests of Reading in a Second Language , 2001 .

[19]  Grant Henning,et al.  A PRELIMINARY STUDY OF THE NATURE OF COMMUNICATIVE COMPETENCE , 1992 .

[20]  F. Vijver,et al.  The incomplete equivalence of the paper-and-pencil and computerized versions of the General Aptitude Test Battery , 1994 .

[21]  James H. Martin,et al.  Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition , 2000 .

[22]  F. Drasgow,et al.  Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis. , 1993 .

[23]  Craig N. Mills,et al.  FIELD TEST OF A COMPUTER-BASED GRE GENERAL TEST , 1993 .

[24]  J. Frommer,et al.  Computer-assisted language learning and testing: Research issues and practice , 1991 .

[25]  R. P. McDonald,et al.  Structural Equations with Latent Variables , 1989 .

[26]  Lyle F. Bachman,et al.  THE CONSTRUCT VALIDATION OF THE FSI ORAL INTERVIEW1 , 1981 .

[27]  Jaeyool Boo,et al.  Computerized and Paper-and-Pencil Versions of the Rosenberg Self-Esteem Scale: A Comparison of Psychometric Features and Respondent Preferences , 2001 .

[28]  Alan C. Bugbee,et al.  The Equivalence of Paper-and-Pencil and Computer-Based Testing. , 1996 .

[29]  Geoff Brindley,et al.  Interfaces between Second Language Acquisition and Language Testing Research: Describing language development? Rating scales and SLA , 1999 .

[30]  Carol A. Chapelle,et al.  Computer applications in second language acquisition : foundations for teaching, testing and research , 2001 .

[31]  Joan Jamieson,et al.  COMPUTER FAMILIARITY AMONG TOEFL EXAMINEES , 1998 .

[32]  William A. Sands,et al.  Computerized adaptive testing: From inquiry to operation. , 1997 .

[33]  R. J. Mokken,et al.  Handbook of modern item response theory , 1997 .

[34]  Mark D. Reckase,et al.  TECHNICAL GUIDELINES FOR ASSESSING COMPUTERIZED ADAPTIVE TESTS , 1984 .

[35]  James Dean Brown,et al.  Computers in language testing: Present research and some future directions , 1997 .

[36]  Carol A. Chapelle,et al.  VALIDITY IN LANGUAGE ASSESSMENT , 1999, Annual Review of Applied Linguistics.

[37]  Howard Wainer,et al.  Computerized Adaptive Testing: A Primer , 2000 .

[38]  Patricia A. Dunkel,et al.  Considerations in Developing or Using Second/Foreign Language Proficiency Computer-Adaptive Tests , 1999 .

[39]  R. Hambleton,et al.  Fundamentals of Item Response Theory , 1991 .

[40]  Glenn Fulcher,et al.  Computerizing an English language placement test , 1999 .