Collecting fluency corrections for spoken learner English

We present crowdsourced collection of error annotations for transcriptions of spoken learner English. Our emphasis in data collection is on fluency corrections, a more complete correction than has traditionally been aimed for in grammatical error correction research (GEC). Fluency corrections require improvements to the text, taking discourse and utterance level semantics into account: the result is a more naturalistic, holistic version of the original. We propose that this shifted emphasis be reflected in a new name for the task: ‘holistic error correction’ (HEC). We analyse crowdworker behaviour in HEC and conclude that the method is useful with certain amendments for future work.

[1]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[2]  Hwee Tou Ng,et al.  Building a Large Annotated Corpus of Learner English: The NUS Corpus of Learner English , 2013, BEA@NAACL-HLT.

[3]  Brendan T. O'Connor,et al.  Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks , 2008, EMNLP.

[4]  Martin Chodorow,et al.  Rethinking Grammatical Error Annotation and Evaluation with the Amazon Mechanical Turk , 2010 .

[5]  Rachele De Felice,et al.  A Classifier-Based Approach to Preposition and Determiner Error Correction in L2 English , 2008, COLING.

[6]  Esther Thelen,et al.  U-Shaped Changes in Behavior: A Dynamic Systems Perspective , 2004 .

[7]  Janet E. Cahn,et al.  A Psychological Model of Grounding and Repair in Dialog , 1999 .

[8]  Ted Briscoe,et al.  Candidate re-ranking for SMT-based grammatical error correction , 2016, BEA@NAACL-HLT.

[9]  Michael Gamon,et al.  Correcting ESL Errors Using Phrasal SMT Techniques , 2006, ACL.

[10]  Helen Yannakoudakis,et al.  Developing and testing a self-assessment and tutoring system , 2013, BEA@NAACL-HLT.

[11]  Hwee Tou Ng,et al.  System Combination for Grammatical Error Correction , 2014, EMNLP.

[12]  Roger Levy,et al.  Automated Whole Sentence Grammar Correction Using a Noisy Channel Model , 2011, ACL.

[13]  Hwee Tou Ng,et al.  How Far are We from Fully Automatic High Quality Grammatical Error Correction? , 2015, ACL.

[14]  Chris Callison-Burch,et al.  Cheap, Fast and Good Enough: Automatic Speech Recognition with Non-Expert Transcription , 2010, NAACL.

[15]  Paula Buttery,et al.  Automated speech-unit delimitation in spoken learner English , 2016, COLING.

[16]  Nitin Madnani,et al.  They Can Help: Using Crowdsourcing to Improve the Evaluation of Grammatical Error Detection Systems , 2011, ACL.

[17]  H. H. Clark,et al.  Collaborating on contributions to conversations , 1987 .

[18]  Dan Roth,et al.  Grammatical Error Correction: Machine Translation and Classifiers , 2016, ACL.

[19]  D Nicholls,et al.  The Cambridge Learner Corpus-Error coding and analysis , 1999 .

[20]  Matt Post,et al.  Reassessing the Goals of Grammatical Error Correction: Fluency Instead of Grammaticality , 2016, TACL.

[21]  Mark J. F. Gales,et al.  Improving multiple-crowd-sourced transcriptions using a speech recogniser , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[22]  Matthew G. Snover,et al.  A Study of Translation Edit Rate with Targeted Human Annotation , 2006, AMTA.

[23]  Marcin Junczys-Dowmunt,et al.  Phrase-based Machine Translation is State-of-the-Art for Automatic Grammatical Error Correction , 2016, EMNLP.

[24]  Ralph Weischedel,et al.  A STUDY OF TRANSLATION ERROR RATE WITH TARGETED HUMAN ANNOTATION , 2005 .

[25]  M. Pickering,et al.  Toward a mechanistic psychology of dialogue , 2004, Behavioral and Brain Sciences.

[26]  Ted Briscoe,et al.  Grammatical error correction using neural machine translation , 2016, NAACL.

[27]  A. Acquisti,et al.  Beyond the Turk: Alternative Platforms for Crowdsourcing Behavioral Research , 2016 .

[28]  Hwee Tou Ng,et al.  A Beam-Search Decoder for Grammatical Error Correction , 2012, EMNLP.

[29]  Alexandra A. Cleland,et al.  Syntactic alignment and participant role in dialogue , 2007, Cognition.

[30]  Helen Yannakoudakis,et al.  A New Dataset and Method for Automatically Grading ESOL Texts , 2011, ACL.

[31]  Raymond Hendy Susanto,et al.  The CoNLL-2014 Shared Task on Grammatical Error Correction , 2014 .