Lexical Chaining for Measuring Discourse Coherence Quality in Test-taker Essays

This paper presents an investigation of lexical chaining (Morris and Hirst, 1991) for measuring discourse coherence quality in test-taker essays. We hypothesize that attributes of lexical chains, as well as interactions between lexical chains and explicit discourse elements, can be harnessed for representing coherence. Our experiments reveal that performance achieved by our new lexical chain features is better than that of previous discourse features used for this task, and that the best system performance is achieved when combining lexical chaining features with complementary discourse features, such as those provided by a discourse parser based on rhetorical structure theory, and features that reflect errors in grammar, word usage, and mechanics.

[1]  Graeme Hirst,et al.  Lexical chains as representations of context for the detection and correction of malapropisms , 1995 .

[2]  Daniel Marcu,et al.  Evaluating Multiple Aspects of Coherence in Student Essays , 2004, NAACL.

[3]  Dan Klein,et al.  An Empirical Investigation of Statistical Significance in NLP , 2012, EMNLP.

[4]  Hwee Tou Ng,et al.  Automatically Evaluating Text Coherence Using Discourse Relations , 2011, ACL.

[5]  Dan I. Moldovan,et al.  Lexical Chains for Question Answering , 2002, COLING.

[6]  William C. Mann,et al.  Rhetorical Structure Theory: Toward a functional theory of text organization , 1988 .

[7]  Mirella Lapata,et al.  Modeling Local Coherence: An Entity-Based Approach , 2005, ACL.

[8]  Michael Halliday,et al.  Cohesion in English , 1976 .

[9]  L. Faigley,et al.  Coherence, Cohesion, and Writing Quality , 1981, College Composition & Communication.

[10]  Livio Robaldo,et al.  The Penn Discourse TreeBank 2.0. , 2008, LREC.

[11]  Peter W. Foltz,et al.  The Measurement of Textual Coherence with Latent Semantic Analysis. , 1998 .

[12]  Graeme Hirst,et al.  Lexical Cohesion Computed by Thesaural relations as an indicator of the structure of text , 1991, CL.

[13]  John Sabatini,et al.  THE LANGUAGE MUSESM SYSTEM: LINGUISTICALLY FOCUSED INSTRUCTIONAL AUTHORING , 2012 .

[14]  Christiane Fellbaum,et al.  Lexical Chains as Representations of Context for the Detection and Correction of Malapropisms , 1998 .

[15]  Jerry R. Hobbs.  Coherence and Coreference , 1979, Cogn. Sci..

[16]  Nicola Stokes,et al.  Spoken and Written News Story Segmentation Using Lexical Chains , 2003, NAACL.

[17]  Charles A. Perfetti,et al.  Discourse Comprehension and Sources of Individual Differences. , 1977 .

[18]  Daniel Marcu,et al.  Discourse Generation Using Utility-Trained Coherence Models , 2006, ACL.

[19]  Dekang Lin,et al.  Automatic Retrieval and Clustering of Similar Words , 1998, ACL.

[20]  Lijun Feng,et al.  Cognitively Motivated Features for Readability Assessment , 2009, EACL.

[21]  Micha Elsner,et al.  A Unified Local and Global Model for Discourse Coherence , 2007, NAACL.

[22]  Regina Barzilay,et al.  Using Lexical Chains for Text Summarization , 1997 .

[23]  Jill Burstein,et al.  Handbook of Automated Essay Evaluation Current Applications and New Directions , 2018 .

[24]  Charles R. Fletcher,et al.  Investigations of inferential processes in reading: A theoretical and methodological integration , 1993 .

[25]  M. Sherwood-Smith,et al.  Lexical chains for topic tracking , 2002, IEEE International Conference on Systems, Man and Cybernetics.

[26]  Paul Deane,et al.  On the relation between automated essay scoring and modern views of the writing construct , 2013 .

[27]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[28]  J. Carthy,et al.  TOPIC DETECTION , A NEW APPLICATION FOR LEXICAL CHAINING ? , 2000 .

[29]  Michael Flor,et al.  Lexical Tightness and Text Complexity , 2013 .

[30]  Martin Chodorow,et al.  Holistic Discourse Coherence Annotation for Noisy Essay Writing , 2013, Dialogue Discourse.

[31]  Micha Elsner,et al.  Coreference-inspired Coherence Modeling , 2008, ACL.

[32]  Jill Burstein,et al.  AUTOMATED ESSAY SCORING WITH E‐RATER® V.2.0 , 2004 .

[33]  Scott Weinstein,et al.  Centering: A Framework for Modeling the Local Coherence of Discourse , 1995, CL.

[34]  Vasile Rus,et al.  Automated Detection of Local Coherence in Short Argumentative Essays Based on Centering Theory , 2012, CICLing.

[35]  Brent Bridgeman,et al.  Comparison of Human and Machine Scoring of Essays: Differences by Gender, Ethnicity, and Country , 2012 .

[36]  Jacob Cohen,et al.  Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. , 1968 .

[37]  Rashmi Prasad,et al.  The Penn Discourse Treebank , 2004, LREC.

[38]  Ani Nenkova,et al.  Revisiting Readability: A Unified Framework for Predicting Text Quality , 2008, EMNLP.

[39]  P. White,et al.  Detecting breakdowns in local coherence in the writing of Chinese English learners , 2012, J. Comput. Assist. Learn..

[40]  Karen Kukich,et al.  Automated Evaluation of Coherence in Student Essays , .

[41]  Martin Chodorow,et al.  Enriching Automated Essay Scoring Using Discourse Marking , 2001 .

[42]  John Sabatini,et al.  Measuring up: Advances in How We Assess Reading Ability. , 2012 .

[43]  Ilyas Cicekli,et al.  Using lexical chains for keyword extraction , 2007, Inf. Process. Manag..

[44]  Alden J. Moe.  Cohesion, Coherence, and the Comprehension of Text. , 1979 .