When it's all piling up: investigating error propagation in an NLP pipeline

We present an analysis of a high-level semantic task, the construction of cross-document event timelines from SemEval 2015 Task 4: TimeLine, to trace down errors to the components of our pipeline system. Event timeline extraction requires many different Natural Language Processing tasks among which entity and event detection, coreference resolution and semantic-role-labeling are pivotal. These tasks yet depend on other low-level analysis. This paper shows where errors come from and whether they are propagated through the different layers. We also show that performance of each of the subtasks is still insufficient for the complex task considered. Finally, we observe that there is not enough semantics and inferencing within the standard NLP techniques to perform well.

[1]  Heeyoung Lee,et al.  Stanford’s Multi-Pass Sieve Coreference Resolution System at the CoNLL-2011 Shared Task , 2011, CoNLL Shared Task.

[2]  Daniel Gildea,et al.  The Proposition Bank: An Annotated Corpus of Semantic Roles , 2005, CL.

[3]  Erik F. Tjong Kim Sang,et al.  Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition , 2003, CoNLL.

[4]  Steven Bethard,et al.  A Synchronous Context Free Grammar for Time Normalization , 2013, EMNLP.

[5]  Joyce Yue Chai,et al.  The Role of Implicit Argumentation in Nominal SRL , 2009, HLT-NAACL.

[6]  Renata Vieira,et al.  A Corpus-based Investigation of Definite Description Use , 1997, CL.

[7]  Josef Ruppenhofer,et al.  Beyond sentence-level semantic role labeling: linking argument structures in discourse , 2013, Lang. Resour. Evaluation.

[8]  Christopher D. Manning.  Part-of-Speech Tagging from 97% to 100%: Is It Time for Some Linguistics? , 2011, CICLing.

[9]  Erik F. Tjong Kim Sang,et al.  Introduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity Recognition , 2002, CoNLL.

[10]  Antske Fokkens,et al.  NAF and GAF: Linking Linguistic Annotations , 2014 .

[11]  James Pustejovsky,et al.  Increasing Informativeness in Temporal Annotation , 2011, Linguistic Annotation Workshop.

[12]  Christiane Fellbaum,et al.  On the Role of Lexical and World Knowledge in RTE3 , 2007, ACL-PASCAL@ACL.

[13]  James Pustejovsky,et al.  SemEval-2013 Task 1: TempEval-3: Evaluating Time Expressions, Events, and Temporal Relations , 2013, *SEMEVAL.

[14]  Tommaso Caselli,et al.  SPINOZA_VU: An NLP Pipeline for Cross Document TimeLines , 2015, SemEval@NAACL-HLT.

[15]  Egoitz Laparra,et al.  Annotated Data , version 2 Deliverable D 3 . 3 . 2 Version DRAFT , 2015 .

[16]  Joyce Yue Chai,et al.  Semantic Role Labeling of Implicit Arguments for Nominal Predicates , 2012, CL.

[17]  Pierre Nugues,et al.  A High-Performance Syntactic and Semantic Dependency Parser , 2010, COLING.

[18]  Eneko Agirre,et al.  SemEval-2015 Task 4: TimeLine: Cross-Document Event Ordering , 2015, *SEMEVAL.