The Impact of Automated Essay Scoring on Writing Outcomes

This study was an expanded replication of an earlier endeavor (Shermis, Burstein, & Bliss, 2004) to document the writing outcomes associated with automated essay scoring. The focus of the current study was on determining whether exposure to multiple writing prompts facilitated writing production variables (Essay Score, Essay Length, and Number of Unique Words) and decreased writing errors (Grammar, Usage, Mechanics, Style, Organization & Development) over time. The impacts of these variables were examined in analyses of 11,685 essays written by 2,017 students at four grade levels (grades 6-8, 10). The essays, written in response to seven different prompts, were scored by automated essay scoring. The results showed significant differences across the four grades and over time for each of the eight outcome variables. Peak essay performance occurred with 8 graders who also displayed the highest reduction of both domain errors. Specific types of error reduction were differentially associated with grade level. The implications of the results for future research incorporating writing genre are discussed.

[1]  E. B. Page,et al.  The Computer Moves into Essay Grading: Updating the Ancient Test. , 1995 .

[2]  Amos Van Gelderen,et al.  Elementary Students' Skills in Revising Integrating Quantitative and Qualitative Analysis , 1997 .

[3]  Yigal Attali,et al.  Exploring the Feedback and Revision Features of Criterion , 2004 .

[4]  Donnell Hunter,et al.  Writing with power , 1981 .

[5]  Jill Burstein,et al.  The E-rater® scoring engine: Automated essay scoring with natural language processing. , 2003 .

[6]  Bryan A. Bardine,et al.  Beyond the Red Pen: Clarifying Our Role in the Response Process , 2000 .

[7]  Richard H. Haswell,et al.  Machine Scoring of Student Essays , 2006 .

[8]  N. Sommers.  Revision Strategies of Student Writers and Experienced Adult Writers , 1980, College Composition & Communication.

[9]  Richard Straub.  The Concept of Control in Teacher Response: Defining the Varieties of "Directive" and "Facilitative" Commentary , 1996 .

[10]  Martin Chodorow,et al.  C-rater: Automated Scoring of Short-Answer Questions , 2003, Comput. Humanit..

[11]  Kyle Perkins,et al.  On the Use of Composition Scoring Techniques, Objective Measures, and Objective Tests to Evaluate ESL Writing Ability , 1983 .

[12]  Jill Burstein,et al.  AUTOMATED ESSAY SCORING WITH E‐RATER® V.2.0 , 2004 .

[13]  Summer Smith.  The Genre of the End Comment: Conventions in Teacher Responses to Student Writing. , 1997 .

[14]  Yigal Attali.  ON-THE-FLY CUSTOMIZATION OF AUTOMATED ESSAY SCORING , 2007 .

[15]  Dana R. Ferris,et al.  Response To Student Writing: Implications for Second Language Students , 2003 .

[16]  Liz Hamp-Lyons,et al.  Rating Nonnative Writing: The Trouble with Holistic Scoring. , 1995 .

[17]  Daniel Marcu,et al.  A Machine Learning Approach for Identification Thesis and Conclusion Statements in Student Essays , 2003, Comput. Humanit..

[18]  Martin Chodorow,et al.  CriterionSM Online Essay Evaluation: An Application for Automated Evaluation of Student Essays , 2003, IAAI.

[19]  T. Keith,et al.  Trait Ratings for Automated Essay Grading , 2002 .

[20]  Robert P. Yagelski,et al.  The Role of Classroom Context in the Revision Strategies of Student Writers , 1995, Research in the Teaching of English.

[21]  G. Molenberghs,et al.  Linear Mixed Models for Longitudinal Data , 2001 .