Evaluating Argumentative and Narrative Essays using Graphs

This work investigates whether the development of ideas in writing can be captured by graph properties derived from the text. Focusing on student essays, we represent the essay as a graph, and encode a variety of graph properties including PageRank as features for modeling essay scores related to quality of development. We demonstrate that our approach improves on a state-of-the-art system on the task of holistic scoring of persuasive essays and on the task of scoring narrative essays along the development dimension.

[1]  Hwee Tou Ng,et al.  Automatically Evaluating Text Coherence Using Discourse Relations , 2011, ACL.

[2]  Graeme Hirst,et al.  Lexical Cohesion Computed by Thesaural relations as an indicator of the structure of text , 1991, CL.

[3]  Swapna Somasundaran,et al.  Automated Scoring of Picture-based Story Narration , 2015, BEA@NAACL-HLT.

[4]  Rohit J. Kate,et al.  Learning to Predict Readability using Diverse Linguistic Features , 2010, COLING.

[5]  Manfred Stede,et al.  Joint prediction in MST-style discourse parsing for argumentation mining , 2015, EMNLP.

[6]  Noura Farra,et al.  Scoring Persuasive Essays Using Opinions and their Targets , 2015, BEA@NAACL-HLT.

[7]  Keelan Evanini,et al.  Automated speech scoring for non-native middle school students with multiple task types , 2013, INTERSPEECH.

[8]  Vasile Rus,et al.  Automated Detection of Local Coherence in Short Argumentative Essays Based on Centering Theory , 2012, CICLing.

[9]  Swapna Somasundaran,et al.  Lexical Chaining for Measuring Discourse Coherence Quality in Test-taker Essays , 2014, COLING.

[10]  Jakob Grue Simonsen,et al.  Entropy and Graph Based Modelling of Document Coherence using Discourse Entities: An Application to IR , 2015, ICTIR.

[11]  Sergey Brin,et al.  The Anatomy of a Large-Scale Hypertextual Web Search Engine , 1998, Comput. Networks.

[12]  Daniel Marcu,et al.  Finding the WRITE Stuff: Automatic Identification of Discourse Structure in Student Essays , 2003, IEEE Intell. Syst..

[13]  Dan Klein,et al.  An Empirical Investigation of Statistical Significance in NLP , 2012, EMNLP.

[14]  Jacob Cohen,et al.  Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. , 1968 .

[15]  Marti A. Hearst.  Text Tiling: Segmenting Text into Multi-paragraph Subtopic Passages , 1997, CL.

[16]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[17]  Micha Elsner,et al.  Extending the Entity Grid with Entity-Specific Features , 2011, ACL.

[18]  Jill Burstein,et al.  AUTOMATED ESSAY SCORING WITH E‐RATER® V.2.0 , 2004 .

[19]  Graeme Hirst,et al.  The Impact of Deep Hierarchical Discourse Structures in the Evaluation of Text Coherence , 2014, COLING.

[20]  Camille Guinaudeau,et al.  Graph-based Local Coherence Modeling , 2013, ACL.

[21]  Michael Strube,et al.  Graph-based Coherence Modeling For Assessing Readability , 2015, *SEMEVAL.

[22]  Mathieu Bastian,et al.  Gephi: An Open Source Software for Exploring and Manipulating Networks , 2009, ICWSM.

[23]  Beata Beigman Klebanov,et al.  Applying Argumentation Schemes for Essay Scoring , 2014, ArgMining@ACL.

[24]  Ani Nenkova,et al.  Revisiting Readability: A Unified Framework for Predicting Text Quality , 2008, EMNLP.

[25]  Lijun Feng,et al.  A Comparison of Features for Automatic Readability Assessment , 2010, COLING.

[26]  Yiu-Kai Ng,et al.  ReadAid: A Robust and Fully-Automated Readability Assessment Tool , 2011, 2011 IEEE 23rd International Conference on Tools with Artificial Intelligence.

[27]  Diane J. Litman,et al.  Incorporating Coherence of Topics as a Criterion in Automatic Response-to-Text Assessment of the Organization of Writing , 2015, BEA@NAACL-HLT.

[28]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[29]  Alphie G. Garing.  Coherence in the Argumentative Essays of First Year College of Liberal Arts Students at De La Salle University , 2014 .

[30]  Yang Liu,et al.  Using Latent Dirichlet Allocation for Child Narrative Analysis , 2013, BioNLP@ACL.

[31]  Mirella Lapata,et al.  Modeling Local Coherence: An Entity-Based Approach , 2005, ACL.

[32]  Ying Zhang,et al.  Interpreting BLEU/NIST Scores: How Much Improvement do We Need to Have a Better System? , 2004, LREC.

[33]  Danielle S. McNamara,et al.  Visualizing Topic Flow in Students' Essays , 2011, J. Educ. Technol. Soc..

[34]  Iryna Gurevych,et al.  Identifying Argumentative Discourse Structures in Persuasive Essays , 2014, EMNLP.

[35]  Micha Elsner,et al.  Disentangling Chat with Local Coherence Models , 2011, ACL.

[36]  Gang Sun,et al.  A Graph-based Readability Assessment Method using Word Coupling , 2015, EMNLP.

[37]  Debanjan Ghosh,et al.  Coarse-grained Argumentation Features for Scoring Persuasive Essays , 2016, ACL.

[38]  Joel R. Tetreault,et al.  Using Entity-Based Features to Model Coherence in Student Essays , 2010, HLT-NAACL.

[39]  Iryna Gurevych,et al.  Annotating Argument Components and Relations in Persuasive Essays , 2014, COLING.

[40]  Jill Burstein,et al.  Handbook of Automated Essay Evaluation Current Applications and New Directions , 2018 .

[41]  Karen Kukich,et al.  Evaluation of text coherence for electronic essay scoring systems , 2004, Natural Language Engineering.