A Corpus for Argumentative Writing Support in German

In this paper, we present a novel annotation approach to capture claims and premises of arguments and their relations in student-written persuasive peer reviews on business models in German language. We propose an annotation scheme based on annotation guidelines that allows to model claims and premises as well as support and attack relations for capturing the structure of argumentative discourse in student-written peer reviews. We conduct an annotation study with three annotators on 50 persuasive essays to evaluate our annotation scheme. The obtained inter-rater agreement of $\alpha=0.57$ for argument components and $\alpha=0.49$ for argumentative relations indicates that the proposed annotation scheme successfully guides annotators to moderate agreement. Finally, we present our freely available corpus of 1,000 persuasive student-written peer reviews on business models and our annotation guidelines to encourage future research on the design and development of argumentative writing support systems for students.

[1]  Iryna Gurevych,et al.  Annotating Argument Components and Relations in Persuasive Essays , 2014, COLING.

[2]  J. Fleiss Measuring nominal scale agreement among many raters. , 1971 .

[3]  Thiemo Wambsganss,et al.  Towards Designing an Adaptive Argumentation Learning Tool , 2019, ICIS.

[4]  Matthias Söllner,et al.  Insights into Using IT-Based Peer Feedback to Practice the Students Providing Feedback Skill , 2019, HICSS.

[5]  Chris Reed,et al.  Argumentation Schemes , 2008 .

[6]  Iryna Gurevych,et al.  Linking the Thoughts: Analysis of Argumentation Structures in Scientific Publications , 2015, ArgMining@HLT-NAACL.

[7]  Iryna Gurevych,et al.  On the Role of Discourse Markers for Discriminating Claims and Premises in Argumentative Discourse , 2015, EMNLP.

[8]  Helen Yannakoudakis,et al.  A New Dataset and Method for Automatically Grading ESOL Texts , 2011, ACL.

[9]  Serena Villata,et al.  Towards a Benchmark of Natural Language Arguments , 2014, NMR 2014.

[10]  F. H. Eemeren,et al.  Argumentation, Communication, and Fallacies: A Pragma-dialectical Perspective , 1992 .

[11]  Siegfried Handschuh,et al.  AL: An Adaptive Learning Support System for Argumentation Skills , 2020, CHI.

[12]  Noam Slonim,et al.  A Benchmark Dataset for Automatic Detection of Claims and Evidence in the Context of Controversial Topics , 2014, ArgMining@ACL.

[13]  A. Tchantchane,et al.  Dealing with Large Classes: A Real Challenge , 2010 .

[14]  Katsuhide Fujita,et al.  Annotating and Analyzing Semantic Role of Elementary Units and Relations in Online Persuasive Arguments , 2019, ACL.

[15]  Vangelis Karkaletsis,et al.  Argument Extraction from News , 2015, ArgMining@HLT-NAACL.

[16]  Larry Ambrose,et al.  The power of feedback. , 2002, Healthcare executive.

[17]  Vincent Ng,et al.  End-to-End Argumentation Mining in Student Essays , 2016, NAACL.

[18]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[19]  James B. Freeman,et al.  Argument Structure and Disciplinary Perspective , 2001 .

[20]  Paolo Torroni,et al.  Argumentation Mining , 2016, ACM Trans. Internet Techn..

[21]  Martin Chodorow,et al.  TOEFL11: A CORPUS OF NON‐NATIVE ENGLISH , 2013 .

[22]  Thiemo Wambsganss,et al.  Towards a Taxonomy of Text Mining Features , 2019, ECIS.

[23]  Silvie Cinková,et al.  Managing Uncertainty in Semantic Tagging , 2012, EACL.

[24]  Chris Reed,et al.  Argument Mining: A Survey , 2020, Computational Linguistics.

[25]  Eyal Shnarch,et al.  Corpus Wide Argument Mining - a Working Solution , 2019, AAAI.

[26]  Kevin D. Ashley,et al.  Toward constructing evidence-based legal arguments using legal decision documents and machine learning , 2013, ICAIL.

[27]  Douglas Walton,et al.  A classification system for argumentation schemes , 2015, Argument Comput..

[28]  D. Walton Argumentation Schemes for Presumptive Reasoning , 1995 .

[29]  Brian Ecker,et al.  Internet Argument Corpus 2.0: An SQL schema for Dialogic Social Media and the Corpora to go with it , 2016, LREC.

[30]  K. Howells The future of education and skills: education 2030: the future we want , 2018 .

[31]  Iryna Gurevych,et al.  Exploiting Debate Portals for Semi-Supervised Argumentation Mining in User-Generated Web Discourse , 2015, EMNLP.

[32]  Vincent Ng,et al.  Give Me More Feedback: Annotating Argument Persuasiveness and Related Attributes in Student Essays , 2018, ACL.

[33]  Diane J. Litman,et al.  Argument Mining for Improving the Automated Scoring of Persuasive Essays , 2018, AAAI.

[34]  Marie-Francine Moens,et al.  Language Resources for Studying Argument , 2008, LREC.

[35]  Robert Östling,et al.  Automated Essay Scoring for Swedish , 2013, BEA@NAACL-HLT.

[36]  Sampo Pyysalo,et al.  brat: a Web-based Tool for NLP-Assisted Text Annotation , 2012, EACL.

[37]  Janyce Wiebe,et al.  MPQA 3.0: An Entity/Event-Level Sentiment Corpus , 2015, NAACL.

[38]  Thiemo Wambsganss,et al.  Unlocking Transfer Learning in Argumentation Mining: A Domain-Independent Modelling Approach , 2020, Wirtschaftsinformatik.

[39]  Robert E. Mercer,et al.  An automated method to build a corpus of rhetorically-classified sentences in biomedical texts , 2014, ArgMining@ACL.

[40]  Jan Marco Leimeister,et al.  Future Work and Enterprise Systems , 2018, Bus. Inf. Syst. Eng..

[41]  Marie-Francine Moens,et al.  Study on the Structure of Argumentation in Case Law , 2008, JURIX.

[42]  P. Black,et al.  Developing the theory of formative assessment , 2009 .

[43]  Klaus krippendorff,et al.  Measuring the Reliability of Qualitative Text Analysis Data , 2004 .

[44]  Benno Stein,et al.  A Review Corpus for Argumentation Analysis , 2014, CICLing.

[45]  Iryna Gurevych,et al.  Recognizing Insufficiently Supported Arguments in Argumentative Essays , 2017, EACL.

[46]  WooSoo Park The New Rhetoric , 2011 .

[47]  Raquel Mochales Palau,et al.  Creating an argumentation corpus: do theories apply to real arguments?: a case study on the legal argumentation of the ECHR , 2009, ICAIL.

[48]  Vincent Ng,et al.  Learning to Give Feedback: Modeling Attributes Affecting Argument Persuasiveness in Student Essays , 2018, IJCAI.

[49]  Wolfgang Ziegler,et al.  An Introduction To Reasoning , 2016 .

[50]  Iryna Gurevych,et al.  Identifying Argumentative Discourse Structures in Persuasive Essays , 2014, EMNLP.

[51]  Marie-Francine Moens,et al.  Argumentation mining: the detection, classification and structure of arguments in text , 2009, ICAIL.

[52]  D. Kuhn Thinking as Argument , 1992 .

[53]  F. Fischer,et al.  A framework to analyze argumentative knowledge construction in computer-supported collaborative learning , 2006, Comput. Educ..

[54]  L. S. Vygotskiĭ,et al.  Mind in society : the development of higher psychological processes , 1978 .

[55]  J. Pollock Cognitive Carpentry: A Blueprint for How to Build a Person , 1995 .

[56]  Beata Beigman Klebanov,et al.  Applying Argumentation Schemes for Essay Scoring , 2014, ArgMining@ACL.

[57]  Torsten Zesch,et al.  Fine-grained essay scoring of a complex writing task for native speakers , 2017, BEA@EMNLP.

[58]  Iryna Gurevych,et al.  Parsing Argumentation Structures in Persuasive Essays , 2016, CL.

[59]  Fiona Browne,et al.  Applying Kernel Methods to Argumentation Mining , 2012, FLAIRS.

[60]  Serena Villata,et al.  Natural Language Arguments: A Combined Approach , 2012, ECAI.