Detecting breakdowns in local coherence in the writing of Chinese English learners

This paper introduces CTutor, an automated writing evaluation (AWE) tool for detecting breakdowns in local coherence and reports on a study that applies it to the writing of Chinese L2 English learners. The program is based on Centering theory (CT), a theory of local coherence and salience. The principles of CT are first introduced and then the design and function of CTutor are described. The effectiveness and reliability of the program was evaluated in a study that compared performance by CTutor and two human raters on the analysis of local incoherence and provision of revision on learner essays. Intermediate Chinese English as a foreign language learners (n = 52) were divided into two groups: one receiving CTutor feedback and the other receiving feedback from human raters. Learners in both groups completed three essays; each of which involved the submission of a first draft, revision with feedback on local coherence quality and re-submission. Our results from the comparison between CTutor and human experts showed that this software tool is able to detect local coherence breakdowns with moderate accuracy (F1-measure is around 0.4). There was also little difference between participants' responses to CTutor feedback and human feedback in terms of revision behaviour, with both feedback modes resulting in similar revision pattern. Potential use of the program in instructional settings is discussed. © 2012 Wiley Periodicals, Inc.

[1]  Mirella Lapata,et al.  Modeling Local Coherence: An Entity-Based Approach , 2005, ACL.

[2]  Michael Halliday,et al.  Cohesion in English , 1976 .

[3]  Daniel M. Bikel,et al.  Intricacies of Collins’ Parsing Model , 2004, CL.

[4]  Simone Paolo Ponzetto,et al.  Exploiting Semantic Role Labeling, WordNet and Wikipedia for Coreference Resolution , 2006, NAACL.

[5]  Charles A. MacArthur,et al.  Handbook of Writing Research , 2005 .

[6]  Tim Ashwell,et al.  Patterns of Teacher Response to Student Writing in a Multiple-Draft Composition Classroom: Is Content Feedback Followed by Form Feedback the Best Method? , 2000 .

[7]  Thomas S. Kane.  The Oxford Guide to Writing: A Rhetoric and Handbook for College Students , 1983 .

[8]  Ute Knoch,et al.  ‘Little coherence, considerable strain for reader’: A comparison between two rating scales for the assessment of coherence , 2007 .

[9]  Yukie Horiba,et al.  Comprehension processes in L2 reading: Language competence, textual coherence, and inferences , 1996, Studies in Second Language Acquisition.

[10]  Claire Gardent,et al.  Improving Machine Learning Approaches to Coreference Resolution , 2002, ACL.

[11]  Karen Kukich,et al.  The Role of Centering Theory's Rough-Shift in the Teaching and Evaluation of Writing Skills , 2000, ACL.

[12]  Christiane Fellbaum,et al.  Book Reviews: WordNet: An Electronic Lexical Database , 1999, CL.

[13]  Jason E. Albrecht,et al.  Updating a situation model: a memory-based text processing view. , 1998, Journal of experimental psychology. Learning, memory, and cognition.

[14]  Jason E. Albrecht,et al.  Updating a mental model: maintaining both local and global coherence , 1993 .

[15]  Scott Weinstein,et al.  Centering: A Framework for Modeling the Local Coherence of Discourse , 1995, CL.

[16]  Pedro M. Domingos,et al.  Joint Unsupervised Coreference Resolution with Markov Logic , 2008, EMNLP.

[17]  Ann Raimes,et al.  Language Proficiency, Writing Ability, and Composing Strategies: A Study of ESL College Student Writers† , 1987 .

[18]  Richard Power,et al.  Optimizing Referential Coherence in Text Generation , 2004, CL.

[19]  Roger Ratcliff,et al.  Inference during reading. , 1992 .

[20]  Debra L. Long,et al.  Comprehension skill and global coherence: a paradoxical picture of poor comprehenders' abilities. , 2001 .

[21]  Daniel M. Bikel.  A Statistical Model for Parsing and Word-Sense Disambiguation , 2000, EMNLP.

[22]  Carl Pollard,et al.  A Centering Approach to Pronouns , 1987, ACL.

[23]  W. Kintsch,et al.  Are Good Texts Always Better? Interactions of Text Coherence, Background Knowledge, and Levels of Understanding in Learning From Text , 1996 .

[24]  P. Foltz.  Discourse Coherence and LSA , 2007 .

[25]  Andrew McCallum,et al.  First-Order Probabilistic Models for Coreference Resolution , 2007, NAACL.

[26]  Karen Kukich,et al.  Evaluation of text coherence for electronic essay scoring systems , 2004, Natural Language Engineering.

[27]  Barbara Di Eugenio,et al.  Centering: A Parametric Theory and Its Instantiations , 2004, Computational Linguistics.

[28]  Laura Hasler,et al.  Centering Theory for Evaluation of Coherence in Computer-Aided Summaries , 2008, LREC.

[29]  Daniel Marcu,et al.  From Local to Global Coherence: A Bottom-Up Approach to Text Planning , 1997, AAAI/IAAI.

[30]  Ellen F. Prince,et al.  Toward a taxonomy of given-new information , 1981 .

[31]  Peter W. Foltz,et al.  The Measurement of Textual Coherence with Latent Semantic Analysis. , 1998 .

[32]  Betty Bamberg,et al.  What Makes a Text Coherent , 1983 .