Towards Feasible Guidelines for the Annotation of Argument Schemes

The annotation of argument schemes represents an important step for argumentation mining. General guidelines for the annotation of argument schemes, applicable to any topic, are still missing due to the lack of a suitable taxonomy in Argumentation Theory and the need for highly trained expert annotators. We present a set of guidelines for the annotation of argument schemes, taking as a framework the Argumentum Model of Topics (Rigotti and Morasso, 2010; Rigotti, 2009). We show that this approach can contribute to solving the theoretical problems, since it offers a hierarchical and finite taxonomy of argument schemes as well as systematic, linguistically-informed criteria to distinguish various types of argument schemes. We describe a pilot annotation study of 30 persuasive essays using multiple minimally trained non-expert annotators .Our findings from the confusion matrixes pinpoint problematic parts of the guidelines and the underlying annotation of claims and premises. We conduct a second annotation with refined guidelines and trained annotators on the 10 essays which received the lowest agreement initially. A significant improvement of the inter-annotator agreement shows that the annotation of argument schemes requires highly trained annotators and an accurate annotation of argumentative components (premises and claims).

[1]  J. Fleiss Measuring nominal scale agreement among many raters. , 1971 .

[2]  Nancy Green,et al.  Identifying Argumentation Schemes in Genetics Research Articles , 2015, ArgMining@HLT-NAACL.

[3]  Fabrizio Macagno,et al.  Argumentation Schemes and Topical Relations. , 2014 .

[4]  F. H. Eemeren,et al.  Argumentation, Communication, and Fallacies: A Pragma-dialectical Perspective , 1992 .

[5]  Noam Slonim,et al.  A Benchmark Dataset for Automatic Detection of Claims and Evidence in the Context of Controversial Topics , 2014, ArgMining@ACL.

[6]  Elena Musi,et al.  Verbs of Appearance and Argument Schemes: Italian Sembrare as an Argumentative Indicator , 2015 .

[7]  Rudi Palmieri,et al.  Corporate Argumentation in Takeover Bids , 2014 .

[8]  J. Pollock Cognitive Carpentry: A Blueprint for How to Build a Person , 1995 .

[9]  Marc Moens,et al.  Articles Summarizing Scientific Articles: Experiments with Relevance and Rhetorical Status , 2002, CL.

[10]  Marie-Francine Moens,et al.  Argumentation mining: the detection, classification and structure of arguments in text , 2009, ICAIL.

[11]  Claire Grover,et al.  Re-using an Argument Corpus to Aid in the Curation of Social Media Collections , 2014, LREC.

[12]  Eddo Rigotti,et al.  Relevance of Context-bound loci to Topical Potential in the Argumentation Stage , 2007 .

[13]  Rob Grootendorst,et al.  Argumentation: across the lines of discipline. , 1987 .

[14]  Sara Rosenthal,et al.  Detecting Opinionated Claims in Online Discussions , 2012, 2012 IEEE Sixth International Conference on Semantic Computing.

[15]  Chris Reed,et al.  Argumentation Schemes , 2008 .

[16]  M. Kienpointner Chapter 24. Towards a Typology of Argumentative Schemes , 1987 .

[17]  Owen Rambow,et al.  Identifying Justifications in Written Dialogs , 2011, 2011 IEEE Fifth International Conference on Semantic Computing.

[18]  Manfred Stede,et al.  Ranking the annotators: An agreement study on argumentation structure , 2013, LAW@ACL.

[19]  Frans H. van Eemeren,et al.  Argumentative Indicators in Discourse, A Pragma-Dialectical Study , 2007, Argumentation Library.

[20]  Eddo Rigotti,et al.  Whether and How Classical Topics can be Revived Within Contemporary Argumentation Theory , 2009, Pondering on Problems of Argumentation.

[21]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[22]  Scott Jacobs,et al.  Rhetoric and Dialectic from the Standpoint of Normative Pragmatics , 2000 .

[23]  J. A. Blair,et al.  Fundamentals of Argumentation Theory : A Handbook of Historical Backgrounds and Contemporary Developments , 1997 .

[24]  Andrea Rocci,et al.  Modality and argumentative discourse relations: A study of the Italian necessity modal dovere , 2012 .

[25]  Iryna Gurevych,et al.  Identifying Argumentative Discourse Structures in Persuasive Essays , 2014, EMNLP.

[26]  Stefan Decker,et al.  Arguments about deletion: how experience improves the acceptability of arguments in ad-hoc online task groups , 2013, CSCW.

[27]  Graeme Hirst,et al.  Classifying arguments by scheme , 2011, ACL.

[28]  Eddo Rigotti,et al.  Comparing the Argumentum Model of Topics to Other Contemporary Approaches to Argument Schemes: The Procedural and Material Components , 2010 .

[29]  Chris Reed,et al.  On Argumentation Schemes and the Natural Classification of Arguments , 2004 .

[30]  Beata Beigman Klebanov,et al.  Applying Argumentation Schemes for Essay Scoring , 2014, ArgMining@ACL.

[31]  Manfred Stede,et al.  Parallel Discourse Annotations on a Corpus of Short Texts , 2016, LREC.

[32]  Collin F. Baker,et al.  A Frames Approach to Semantic Analysis , 2009 .

[33]  Nina Wacholder,et al.  Analyzing Argumentative Discourse Units in Online Interactions , 2014, ArgMining@ACL.

[34]  Manfred Stede,et al.  From Argument Diagrams to Argumentation Mining in Texts: A Survey , 2013, Int. J. Cogn. Informatics Nat. Intell..

[35]  Douglas Walton,et al.  Using Argumentation Schemes for Argument Extraction: A Bottom-Up Method , 2012, Int. J. Cogn. Informatics Nat. Intell..

[36]  Chris Reed,et al.  Araucaria: Software for Argument Analysis, Diagramming and Representation , 2004, Int. J. Artif. Intell. Tools.

[37]  Serena Villata,et al.  Combining Textual Entailment and Argumentation Theory for Supporting Online Debates Interactions , 2012, ACL.