Formative Essay Feedback Using Predictive Scoring Models

A major component of secondary education is learning to write effectively, a skill which is bolstered by repeated practice with formative guidance. However, providing focused feedback to every student on multiple drafts of each essay throughout the school year is a challenge for even the most dedicated of teachers. This paper first establishes a new ordinal essay scoring model and its state of the art performance compared to recent results in the Automated Essay Scoring field. Extending this model, we describe a method for using prediction on realistic essay variants to give rubric-specific formative feedback to writers. This method is used in Revision Assistant, a deployed data-driven educational product that provides immediate, rubric-specific, sentence-level feedback to students to supplement teacher guidance. We present initial evaluations of this feedback generation, both offline and in deployment.

[1]  Mark D. Shermis,et al.  State-of-the-art automated essay scoring: Competition, results, and future directions from a United States demonstration , 2014 .

[2]  William Wresch,et al.  The Imminence of Grading Essays by Computer-25 Years Later , 1993 .

[3]  M. Chodorow,et al.  BEYOND ESSAY LENGTH: EVALUATING E-RATER®'S PERFORMANCE ON TOEFL® ESSAYS , 2004 .

[4]  Kellie J. Archer,et al.  glmnetcr: An R Package for Ordinal Response Prediction in High-dimensional Data Settings , 2010 .

[5]  Hwee Tou Ng,et al.  A Neural Approach to Automated Essay Scoring , 2016, EMNLP.

[6]  Lawrence M. Rudner,et al.  Automated Essay Scoring Using Bayes' Theorem , 2002 .

[7]  Robin Martin.  Rhetoric of Teacher Comments on Student Writing , 2011 .

[8]  Yue Zhang,et al.  Automatic Features for Essay Scoring – An Empirical Study , 2016, EMNLP.

[9]  Danielle S. McNamara,et al.  Feedback and Revising in an Intelligent Tutoring System for Writing Strategies , 2013, AIED.

[10]  Dan Klein,et al.  Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network , 2003, NAACL.

[11]  Edward S. Epstein,et al.  A Scoring System for Probability Forecasts of Ranked Categories , 1969 .

[12]  Jing Chen,et al.  Building e‐rater® Scoring Models Using Machine Learning Methods , 2016 .

[13]  D. Ferris.  Student Reactions to Teacher Response in Multiple‐Draft Composition Classrooms* , 1995 .

[14]  Hwee Tou Ng,et al.  Flexible Domain Adaptation for Automated Essay Scoring Using Correlated Linear Regression , 2015, EMNLP.

[15]  Helen Yannakoudakis,et al.  Developing and testing a self-assessment and tutoring system , 2013, BEA@NAACL-HLT.

[16]  Jill Burstein,et al.  Automated Essay Scoring : A Cross-disciplinary Perspective , 2003 .

[17]  D. Nicol,et al.  Formative assessment and self‐regulated learning: a model and seven principles of good feedback practice , 2006 .

[18]  A. Agresti.  Analysis of Ordinal Categorical Data , 1985 .

[19]  V. Shute.  Focus on Formative Feedback , 2007 .

[20]  Helen Yannakoudakis,et al.  Automatic Text Scoring Using Neural Networks , 2016, ACL.

[21]  A. Agresti.  Analysis of Ordinal Categorical Data: Agresti/Analysis , 2010 .

[22]  Stephen R. Cole,et al.  Regression models for unconstrained, partially or fully constrained continuation odds ratios. , 2001, International journal of epidemiology.

[23]  Regina Barzilay,et al.  Rationalizing Neural Predictions , 2016, EMNLP.

[24]  D. Ferris.  The Influence of Teacher Commentary on Student Revision , 1997 .

[25]  Semire Dikli,et al.  An Overview of Automated Scoring of Essays. , 2006 .

[26]  Yigal Attali,et al.  Exploring the Feedback and Revision Features of Criterion , 2004 .

[27]  Laura K. Allen,et al.  A Hierarchical Classification Approach to Automated Essay Scoring. , 2015 .

[28]  Danielle S. McNamara,et al.  Developing pedagogically-guided algorithms for intelligent writing feedback , 2013, Int. J. Learn. Technol..

[29]  Torsten Zesch,et al.  Task-Independent Features for Automated Essay Grading , 2015, BEA@NAACL-HLT.

[30]  Ted Briscoe,et al.  Constrained Multi-Task Learning for Automated Essay Scoring , 2016, ACL.

[31]  M. Weaver,et al.  Do students value feedback? Student perceptions of tutors’ written responses , 2006 .

[32]  Hedy McGarrell,et al.  Motivating revision of drafts through formative feedback , 2007 .

[33]  Fan Zhang,et al.  Annotation and Classification of Argumentative Writing Revisions , 2015, BEA@NAACL-HLT.