Trait Ratings for Automated Essay Grading

This study employed an automated grader to evaluate essays, both holistically and with the rating of traits (content, organization, style, mechanics, and creativity) for Webbased student essays serving as placement tests at a large Midwestern university. The authors report the results of two combined experiments, based on random selection from 1,193 essays. In the first experiment, the essays of 807 students were used to create statistical predictions for the essay-grading software. In the second experiment, the ratings from a separate, random sample of 386 essays were used to compare the ratings of six human judges against those generated by the computer. The interjudge correlation of the human raters alone was r = .71. But the interrater reliability of all six judges in combination with computer scoring reached .83. The essay-grading software was an efficient means for evaluating the essays, with a capacity for grading approximately six documents every second. Other potential feedback measures for use in writing courses are also discussed.

[1]  Susy Macqueen,et al.  Validity , 1973, Just Algorithms.

[2]  Richard J. Stiggins,et al.  An Analysis of Published Tests of Writing Proficiency , 1983 .

[3]  Kyle Perkins,et al.  On the Use of Composition Scoring Techniques, Objective Measures, and Objective Tests to Evaluate ESL Writing Ability , 1983 .

[4]  E. White.  Teaching and assessing writing , 1996 .

[5]  R. Linn.  Educational measurement, 3rd ed. , 1989 .

[6]  B. Huot,et al.  Validating holistic scoring for writing assessment : theoretical and empirical foundations , 1993 .

[7]  E. B. Page.  Computer Grading of Student Prose, Using Modern Concepts and Software , 1994 .

[8]  Gale H. Roid,et al.  Patterns of Writing Skills Derived From Cluster Analysis of Direct-Writing Assessments , 1994 .

[9]  Liz Hamp-Lyons,et al.  Rating Nonnative Writing: The Trouble with Holistic Scoring. , 1995 .

[10]  E. B. Page,et al.  The Computer Moves into Essay Grading: Updating the Ancient Test. , 1995 .

[11]  Computer Grading of Essay Traits in Student Writing , 1996 .

[12]  Randy Elliot Bennett,et al.  VALIDITY AND AUTOMATED SCORING: IT'S NOT ONLY THE SCORING , 1997 .

[13]  Mike Brown,et al.  Computerized Adaptive Testing through the World Wide Web. , 1997 .

[14]  Timothy Z. Keith,et al.  Computer Analysis of Student Essays: Finding Trait Differences in Student Profile , 1997 .

[15]  Martin Chodorow,et al.  Computer Analysis of Essay Content for Automated Score Prediction , 1998 .

[16]  Howard R. Mzumara.  VALIDITY OF THE IUPUI PLACEMENT TEST SCORES FOR COURSE PLACEMENT: 1997-1998 , 1998 .

[17]  Identifiers California,et al.  Annual Meeting of the National Council on Measurement in Education , 1998 .

[18]  Randy Elliot Bennett,et al.  Validity and Automad Scoring: It's Not Only the Scoring , 1998 .

[19]  Mark D. Shermis,et al.  The influence of word processing on English placement test results , 2000 .

[20]  Howard R. Mzumara,et al.  On-line Grading of Student Essays: PEG goes on the World Wide Web , 2001 .