In the face of fallible AWE feedback: how do students respond?

Abstract Automated writing evaluation (AWE) systems can provide immediate computer-generated quantitative assessments and qualitative diagnostic feedback on an enormous number of submitted essays. However, limited research attention has been paid to locally designed AWE systems used in English as a foreign language (EFL) classroom contexts. This study explored the precision of feedback provided by a Chinese AWE programme Pigai and students’ uptake of such feedback. Data comprising Pigai feedback, students’ revised submissions, incorporated Pigai suggestions and proportions of ‘good’, ‘neutral’ and ‘bad’ revisions were collected from 210 drafts by 30 undergraduate students in an EFL writing class. Results show that (1) the precision rates of AWE feedback differed across categories of feedback (ie grammar, collocations and mechanics); (2) students were selective in their utilisation of AWE feedback and able to adjust their uptake of AWE suggestions according to the accuracy of the feedback. Survey and interview data demonstrated that the students were cognisant of the strengths and limitations of the AWE system as a source of feedback for improving their writing. These results indicate that AWE feedback can usefully supplement peer and instructor feedback in the EFL writing classroom but cannot replace the latter, given the low precision and accuracy rates of the former in identifying a range of errors.

[1]  Semire Dikli,et al.  The Nature of Automated Essay Scoring Feedback , 2010 .

[2]  Martin Chodorow,et al.  CriterionSM Online Essay Evaluation: An Application for Automated Evaluation of Student Essays , 2003, IAAI.

[3]  Identifiers California,et al.  Annual Meeting of the National Council on Measurement in Education , 1998 .

[4]  Semire Dikli,et al.  Automated Essay Scoring feedback for second language writers: How does it compare to instructor feedback? , 2014 .

[5]  Yigal Attali,et al.  Exploring the Feedback and Revision Features of Criterion , 2004 .

[6]  Richard Badger,et al.  A comparative study of peer and teacher feedback in a Chinese EFL writing class , 2006 .

[7]  Daisuke Kawahara,et al.  The Effect of Corpus Size on Case Frame Acquisition for Discourse Analysis , 2009, NAACL.

[8]  Mark Warschauer,et al.  Utility in a Fallible Tool: A Multi-Site Case Study of Automated Writing Evaluation. , 2010 .

[9]  Elena Cotos,et al.  Validity arguments for diagnostic assessment using automated writing evaluation , 2015 .

[10]  Clyde A. Warden.  EFL Business Writing Behaviors in Differing Feedback Environments , 2000 .

[11]  Goodith White,et al.  A process genre approach to teaching writing , 2000 .

[12]  Charlene Polio,et al.  THE ACCURACY OF COMPUTER-ASSISTED FEEDBACK AND STUDENTS' RESPONSES TO IT , 2015 .

[13]  M. Warschauer,et al.  Learning with Laptops: A Multi-Method Case Study , 2008 .

[14]  John Sinclair,et al.  Corpus, Concordance, Collocation , 1991 .

[15]  Elahe Movahediyan Attar,et al.  The Effects of Teaching Lexical Collocations on Speaking Ability of Iranian EFL Learners , 2013 .

[16]  N. A-R A E H A N,et al.  Detecting errors in English article usage by non-native speakers , 2006 .

[17]  Nadja Nesselhauf.  THE USE OF COLLOCATIONS BY ADVANCED LEARNERS OF ENGLISH AND SOME IMPLICATIONS FOR TEACHING , 2003 .

[18]  M. Farghal,et al.  COLLOCATIONS: A NEGLECTED VARIABLE IN EFL , 1995 .

[19]  John Bitchener,et al.  Written corrective feedback in second language acquisition and writing studies , 2011, Language Teaching.

[20]  Edward M. White,et al.  Teaching and Assessing Writing: Recent Advances in Understanding, Evaluating, and Improving Student Performance. The Jossey-Bass Higher Education Series. , 1985 .

[21]  Chaitanya Ramineni.  Validating Automated Essay Scoring for Online Writing Placement , 2013 .

[22]  Dilin Liu,et al.  L2 vs. L1 Use of Synonymy: An Empirical Study of Synonym Use/Acquisition , 2016 .

[23]  Yi-hsiu Lai,et al.  Which do students prefer to evaluate their essays: Peers or computer program , 2010, Br. J. Educ. Technol..

[24]  Hsien-Chin Liou,et al.  Online synonym materials and concordancing for EFL college writing , 2007 .

[25]  Martin Chodorow,et al.  The Ups and Downs of Preposition Error Detection in ESL Writing , 2008, COLING.

[26]  Lawrence M. Rudner,et al.  An Evaluation of IntelliMetric™ Essay Scoring System , 2006 .

[27]  Dana R. Ferris,et al.  Response To Student Writing: Implications for Second Language Students , 2003 .

[28]  Sara Cushing Weigle.  English as a Second Language Writing and Automated Essay Evaluation , 2013 .

[29]  Evelyn Benson,et al.  The BBI Combinatory Dictionary of English: Your guide to collocations and grammar. , 2010 .

[30]  Antony John Kunnan,et al.  Investigating the Application of Automated Writing Evaluation to Chinese Undergraduate English Majors: A Case Study of WriteToLearn , 2016 .

[31]  Claude Baudoin Iic,et al.  Vocabulary , 2007, Selections from Horace Odes III.

[32]  Nadja Nesselhauf,et al.  Collocations in a Learner Corpus , 2005 .

[33]  Martin Chodorow,et al.  Automated Essay Scoring for Nonnative English Speakers , 1999 .

[34]  Judy F. Chen,et al.  Computer Generated Error Feedback and Writing Process: A Link. , 1997 .

[35]  Chi-Fen Emily Chen,et al.  Beyond the Design of Automated Writing Evaluation: Pedagogical Practices and Perceived Learning Effectiveness in EFL Writing Classes. , 2008 .

[36]  Andrew Klobucar,et al.  Automated Scoring in Context: Rapid Assessment for Placed Students. , 2013 .

[37]  Aek Phakiti,et al.  The effects of computer-generated feedback on the quality of writing , 2014 .

[38]  M. Martin.  Advanced Vocabulary Teaching: The Problem of Synonyms* , 1984 .

[39]  Marjorie T. Davis,et al.  Teaching and Assessing Writing: Recent Advances in Understanding, Evaluating, and Improving Student Performance , 1988 .

[40]  Volker Hegelheimer,et al.  The role of automated writing evaluation holistic scores in the ESL classroom , 2014 .

[41]  Supakorn Phoocharoensil.  Collocational Errors in EFL Learners' Interlanguage , 2011 .

[42]  J. Bahns.  Lexical collocations: a contrastive view , 1993 .

[43]  Stephanie Link,et al.  Automated writing evaluation for formative assessment of second language writing: investigating the accuracy and usefulness of feedback as part of argument-based validation , 2017 .

[44]  Joel R. Tetreault,et al.  The utility of article and preposition error correction systems for English language learners: Feedback and assessment , 2010 .

[45]  Changhua Sun Rich.  The Impact of Online Automated Writing Evaluation: A Case Study from Dalian , 2012 .

[46]  G. Hu.  Developing an EAP Writing Course for Chinese ESL Students , 2007 .

[47]  P. Foltz,et al.  Tracking Student Learning in a State-Wide Implementation of Automated Writing Scoring , 2013 .

[48]  Mark Warschauer,et al.  Automated writing evaluation: defining the classroom research agenda , 2006 .