Effects of marking method and rater experience on ESL essay scores and rater performance

This study examined the effects of marking method and rater experience on ESL (English as a Second Language) essay test scores and rater performance. Each of 31 novice and 29 experienced raters rated a sample of ESL essays both holistically and analytically. Essay scores were analysed using a multi-faceted Rasch model to compare test-takers’ writing ability estimates, inter-rater agreement, and rater severity and self-consistency across marking methods (holistic vs. analytic) and rater groups (novice vs. experienced). Raters tended to be less severe with analytic marking. Holistic marking led to a higher level of inter-rater agreement, while analytic marking led to higher self-consistency. The two rater groups differed in terms of severity, inter-rater agreement and self-consistency. Furthermore, the two marking methods performed differently across the two rater groups. Overall, the findings suggest that the two marking methods might be useful for different assessment purposes, contexts and raters. Implications for ESL writing assessment practice and research are discussed.

[1]  Khaled Barkaoui.  Participants, texts, and processes in second language writing assessment: A narrative review of the literature. , 2007 .

[2]  Khaled Barkaoui,et al.  Explaining ESL essay holistic scores: A multilevel modeling approach , 2010 .

[3]  Nancy Rost Goulden,et al.  Relationship of Analytic and Holistic Methods to Raters' Scores for Speeches. , 1994 .

[4]  Liz Hamp-Lyons,et al.  TOEFL 2000 : writing : composition, community, and assessment , 1997 .

[5]  Uncovering Rater’s Cognitive Processing and Focus Using Think-Aloud Protocols , 2011 .

[6]  Tom Lumley,et al.  Assessing second language writing : the rater's perspective , 2005 .

[7]  Edward W. Wolfe,et al.  STRENGTHENING THE TIES THAT BIND: IMPROVING THE LINKING NETWORK IN SPARSELY CONNECTED RATING DESIGNS , 2000 .

[8]  Nahla Nola Bacha,et al.  Writing Evaluation: What Can Analytic versus Holistic Essay Scoring Tell Us?. , 2001 .

[9]  Bailin Song,et al.  Do English and ESL faculty differ in evaluating the essays of native English-speaking and ESL students? , 1996 .

[10]  Khaled Barkaoui,et al.  Variability in ESL Essay Rating Processes: The Role of the Rating Scale and Rater Experience , 2010 .

[11]  Liz Hamp-Lyons,et al.  Rating Nonnative Writing: The Trouble with Holistic Scoring. , 1995 .

[12]  Sara Cushing Weigle,et al.  Effects of training on raters of ESL compositions , 1994 .

[13]  Alister Cumming,et al.  Expertise in evaluating second language compositions , 1990 .

[14]  M. Vergeer,et al.  The assessment of writing ability: expert readers versus lay readers , 1997 .

[15]  Andy P. Field,et al.  Discovering Statistics Using SPSS , 2000 .

[16]  K. Barkaoui,et al.  Participants, Texts, and Processes in ESL/EFL Essay Tests: A Narrative Review of the Literature , 2007 .

[17]  Is Teaching Experience Necessary for Reliable Scoring of Extended English Questions , 2009 .

[18]  Nancy Rost Goulden,et al.  Theory and vocabulary for communication assessments , 1992 .

[19]  Carol O. Sweedler-Brown.  The Influence of Training and Experience on Holistic Essay Evaluations. , 1985 .

[20]  K. O’Loughlin.  The Assessment of Writing by English and ESL Teachers. , 1994 .

[21]  M. R. Espejo.  Applying the Rasch Model: Fundamental Measurement in the Human Sciences , 2004 .

[22]  Sara Cushing Weigle,et al.  Using FACETS to model rater training effects , 1998 .

[23]  Khaled Barkaoui,et al.  Rating scale impact on EFL essay marking: A mixed-method study , 2007 .

[24]  Michael Ranney,et al.  Cognitive Differences in Proficient and Nonproficient Essay Scorers , 1998 .

[25]  T. McNamara.  Measuring Second Language Performance , 1996 .

[26]  Rob Schoonen,et al.  Generalizability of writing scores: an application of structural equation modeling , 2005 .

[27]  Kyle Perkins,et al.  On the Use of Composition Scoring Techniques, Objective Measures, and Objective Tests to Evaluate ESL Writing Ability , 1983 .