COMPARABILITY OF TOEFL CBT WRITING PROMPTS: RESPONSE MODE ANALYSES

Eighty-three Test of English as a Foreign Language™ (TOEFL®) CBT writing prompts that were administered between July 1998 and August 2000 were examined in order to identify differences in scores that could be attributed to the response mode chosen by examinees (handwritten or word processed). Differences were examined statistically using polytomous logistic regression. An English language ability (ELA) variable was developed from the multiple-choice components of the TOEFL examination and used as a matching variable. Although there was little observed difference in mean writing scores, when examinees were matched on English language ability, small differences were observed in effect sizes consistently favoring the handwritten response mode. The difference favoring the handwritten response mode occurred for all of the writing prompts analyzed; however, the differences for individual writing prompts were small. This difference suggests a general effect for response mode.

[1]  Edward Wolfe,et al.  A Comparison of Word-Processed and Handwritten Essays from a Standardized Writing Assessment. ACT Research Report Series 93-8. , 1993 .

[2]  Robin Porter.  Writing and Word Processing in Year One , 1986 .

[3]  Sara Cushing Weigle,et al.  Using FACETS to model rater training effects , 1998 .

[4]  Timothy R. Miller,et al.  Logistic Regression and Its Use in Detecting Differential Item Functioning in Polytomous Items , 1996 .

[5]  Michael Russell,et al.  Testing Writing on Computers: An Experiment Comparing Student Performance on Tests Conducted via Computer and via Paper-and-Pencil , 1997 .

[6]  T. Lumley.  Assessment criteria in a large-scale writing test: what do they really mean to the raters? , 2002 .

[7]  Edward W. Wolfe,et al.  A Study of Word Processing Experience and its Effects on Student Essay Writing , 1996 .

[8]  David D. Williams,et al.  Learning from Others: Service-Learning in Costa Rica and Indonesia. , 1997 .

[9]  F. Samejima.  Estimation of latent ability using a response pattern of graded scores , 1968 .

[10]  J. Linacre,et al.  Many-facet Rasch measurement , 1994 .

[11]  Mark J. Gierl,et al.  Evaluating Type I Error and Power Rates Using an Effect Size Measure With the Logistic Regression Procedure for DIF Detection , 2001 .

[12]  Michael J. Hannafin,et al.  The Effects of Word Processing on Written Composition , 1987 .

[13]  Alison Green,et al.  Verbal Protocol Analysis in Language Testing Research: A Handbook , 1998 .

[14]  J. Lutz.  A Study of Professional and Experienced Writers Revising and Editing at the Computer and with Pen and Paper , 1987 .

[15]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[16]  Brent Bridgeman,et al.  Comparability of Scores on Word-Processed and Handwritten Essays on the Graduate Management Admissions Test. , 1998 .

[17]  F. Samejima.  Graded Response Model , 1997 .

[18]  Sara Cushing Weigle,et al.  Effects of training on raters of ESL compositions , 1994 .

[19]  Edward W. Wolfe,et al.  AN INVESTIGATION OF THE IMPACT OF COMPOSITION MEDIUM ON THE QUALITY OF TOEFL WRITING SCORES , 2004 .

[20]  B. Zumbo.  A Handbook on the Theory and Methods of Differential Item Functioning (DIF) LOGISTIC REGRESSION MODELING AS A UNITARY FRAMEWORK FOR BINARY AND LIKERT-TYPE (ORDINAL) ITEM SCORES , 1999 .

[21]  Joan Jamieson,et al.  THE RELATIONSHIP BETWEEN COMPUTER FAMILIARITY AND PERFORMANCE ON COMPUTER-BASED TOEFL TEST TASKS , 1998 .

[22]  Marisa Farnum,et al.  WILL THEY THINK LESS OF MY HANDWRITTEN ESSAY IF OTHERS WORD PROCESS THEIRS? EFFECTS ON ESSAY SCORES OF INTERMINGLING HANDWRITTEN AND WORD-PROCESSED ESSAYS , 1992 .

[23]  David W. Hosmer,et al.  Applied Logistic Regression , 1991 .

[24]  K. A. Ericsson,et al.  Protocol analysis: Verbal reports as data, Rev. ed. , 1993 .

[25]  Edward W. Wolfe,et al.  The Impact of Composition Medium on Essay Raters in Foreign Language Testing , 2000 .

[26]  Joan Jamieson,et al.  COMPUTER FAMILIARITY AMONG TOEFL EXAMINEES , 1998 .

[27]  Ruth J. Kurth.  Using Word Processing to Enhance Revision Strategies during Student Writing Activities. , 1987 .

[28]  William Stout,et al.  Simulation Studies of the Effects of Small Sample Size and Studied Item Parameters on SIBTEST and Mantel‐Haenszel Type I Error Performance , 1996 .

[29]  H. Swaminathan,et al.  Detecting Differential Item Functioning Using Logistic Regression Procedures , 1990 .

[30]  Gail E. Hawisher.  The Effects of Word Processing on the Revision Strategies of College Freshmen , 1987 .

[31]  Edward W. Wolfe,et al.  A Comparison of Word-Processed and Handwritten Essays Written for the Test of English as a Foreign Language. , 2000 .