Interoperable Annotation of Events and Event Relations across Domains

This paper presents methodologies for interoperable annotation of events and event relations across different domains, based on notions proposed in prior work. In addition to the interoperability, our annotation scheme supports a wide coverage of events and event relations. We employ the methodologies to annotate events and event relations on Simple Wikipedia articles in 10 different domains. Our analysis demonstrates that the methodologies can allow us to annotate events and event relations in a principled manner against the wide variety of domains. Despite our relatively wide and flexible annotation of events, we achieve high inter-annotator agreement on event annotation. As for event relations, we obtain reasonable inter-annotator agreement. We also provide an analysis of issues on annotation of events and event relations that could lead to annotators’ disagreement.

[1]  Sampo Pyysalo,et al.  brat: a Web-based Tool for NLP-Assisted Text Annotation , 2012, EACL.

[2]  Emmon Bach,et al.  The algebra of events , 1986, The Language of Time - A Reader.

[3]  Lynette Hirschman,et al.  A Model-Theoretic Coreference Scoring Scheme , 1995, MUC.

[4]  Sampo Pyysalo,et al.  Overview of BioNLP’09 Shared Task on Event Extraction , 2009, BioNLP@HLT-NAACL.

[5]  Teruko Mitamura,et al.  Events Detection, Coreference and Sequencing: What's next? Overview of the TAC KBP 2017 Event Track , 2017, TAC.

[6]  S. Pulman,et al.  Aspectual Shift as Type Coercion , 1997 .

[7]  John B. Lowe,et al.  The Berkeley FrameNet Project , 1998, ACL.

[8]  M. Felisa Verdejo,et al.  Events are Not Simple: Identity, Non-Identity, and Quasi-Identity , 2013, EVENTS@NAACL-HLT.

[9]  Breck Baldwin,et al.  Algorithms for Scoring Coreference Chains , 1998 .

[10]  Seth Kulick,et al.  From Light to Rich ERE: Annotation of Entities, Relations, and Events , 2015, EVENTS@HLP-NAACL.

[11]  Roger C. Schank,et al.  Scripts, plans, goals and understanding: an inquiry into human knowledge structures , 1978 .

[12]  Nigel Collier,et al.  PASBio: predicate-argument structures for event extraction in molecular biology , 2004, BMC Bioinformatics.

[13]  Daniel Gildea,et al.  The Proposition Bank: An Annotated Corpus of Semantic Roles , 2005, CL.

[14]  Martha Palmer,et al.  Richer Event Description: Integrating event coreference with temporal, causal and bridging annotation , 2016 .

[15]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[16]  J. Fleiss.  Measuring nominal scale agreement among many raters. , 1971 .

[17]  James F. Allen.  Maintaining knowledge about temporal intervals , 1983, CACM.

[18]  Xiaoqiang Luo,et al.  On Coreference Resolution Performance Metrics , 2005, HLT.

[19]  Mark A. Przybocki,et al.  The Automatic Content Extraction (ACE) Program – Tasks, Data, and Evaluation , 2004, LREC.

[20]  George A. Miller,et al.  Introduction to WordNet: An On-line Lexical Database , 1990 .

[21]  James Pustejovsky,et al.  TimeML: Robust Specification of Event and Temporal Expressions in Text , 2003, New Directions in Question Answering.

[22]  Yukari Yamakawa,et al.  Event Nugget Annotation: Processes and Issues , 2015, EVENTS@HLP-NAACL.

[23]  Jaime G. Carbonell,et al.  The BECauSE Corpus 2.0: Annotating Causality and Overlapping Relations , 2017, LAW@ACL.

[24]  Mitchell P. Marcus,et al.  OntoNotes : A Large Training Corpus for Enhanced Processing , 2017 .

[25]  Piek T. J. M. Vossen,et al.  Using a sledgehammer to crack a nut? Lexical diversity and event coreference resolution , 2014, LREC.

[26]  M. R E C A S E,et al.  BLANC: Implementing the Rand index for coreference evaluation , 2010, Natural Language Engineering.

[27]  Yukari Yamakawa,et al.  Generating Questions and Multiple-Choice Answers using Semantic Analysis of Texts , 2016, COLING.

[28]  James Pustejovsky,et al.  ISO/DIS 24617-1 Language Resources management - semantic annotation framework Part 1 : Time and events , 2009 .

[29]  Mark Steedman,et al.  Temporal Ontology and Temporal Reference , 1988, CL.