Developing Pedagogically-Guided Threshold Algorithms for Intelligent Automated Essay Feedback

Current computer based tools for writing instruction show high scoring accuracy but uncertain instructional efficacy. One explanation is that these systems may not effectively communicate valid and appropriate formative feedback. In this paper, we describe an exploratory method for developing feedback algorithms that are grounded in writing pedagogy. The resulting threshold algorithms are shown to be meaningfully related to essay quality and informative regarding individualized, formative feedback for writers.

[1]  Lawrence M. Rudner,et al.  An Evaluation of IntelliMetric™ Essay Scoring System , 2006 .

[2]  T. Landauer.  Automatic Essay Assessment , 2003 .

[3]  Mark Warschauer,et al.  Automated writing evaluation: defining the classroom research agenda , 2006 .

[4]  Mark Warschauer,et al.  Utility in a Fallible Tool: A Multi-Site Case Study of Automated Writing Evaluation. , 2010 .

[5]  Jill Burstein,et al.  AUTOMATED ESSAY SCORING WITH E‐RATER® V.2.0 , 2004 .

[6]  Thomas Quinlan,et al.  Does Automated Feedback Help Students Learn to Write? , 2010 .

[7]  Michael T. Kane,et al.  Validity Issues for Performance-Based Tests Scored With Computer-Automated Scoring Systems , 2002 .

[8]  Lawrence M. Rudner,et al.  An Evaluation of the IntelliMetric[SM] Essay Scoring System. , 2006 .

[9]  Arthur C. Graesser,et al.  Computational Analyses of Multilevel Discourse Comprehension , 2011, Top. Cogn. Sci..

[10]  V. Shute.  Focus on Formative Feedback , 2008 .

[11]  Jill Burstein,et al.  Automated Essay Scoring : A Cross-disciplinary Perspective , 2003 .

[12]  Danielle S. McNamara,et al.  Predicting Human Scores of Essay Quality Using Computational Indices of Linguistic and Textual Features , 2011, AIED.

[13]  N. Sommers.  Responding to Student Writing , 1982 .

[14]  S. D. L. Paz.  Rubrics Heuristics for Developing Writing Strategies , 2009 .

[15]  Patricia Charlton,et al.  Conference on Artificial Intelligence in Education , 2009 .

[16]  Danielle S. McNamara,et al.  Internal Usability Testing of Automated Essay Feedback in an Intelligent Writing Tutor , 2011, FLAIRS Conference.

[17]  Hedy McGarrell,et al.  Motivating revision of drafts through formative feedback , 2007 .

[18]  R. T. Kellogg,et al.  Improving the writing skills of college students , 2007, Psychonomic bulletin & review.

[19]  Arthur C. Graesser,et al.  The Writing-Pal: Natural Language Algorithms to Support Intelligent Tutoring on Writing Strategies , 2012 .

[20]  Martin Chodorow,et al.  Automated Essay Evaluation: The Criterion Online Writing Service , 2004, AI Mag..

[21]  Arthur C. Graesser,et al.  Coh-Metrix: An automated tool for theoretical and applied natural language processing , 2011 .