SemEval-2015 Task 4: TimeLine: Cross-Document Event Ordering

This paper describes the outcomes of the TimeLine task (Cross-Document Event Ordering), that was organised within the Time and Space track of SemEval-2015. Given a set of documents and a set of target entities, the task consisted of building a timeline for each entity, by detecting, anchoring in time and ordering the events involving that entity. The TimeLine task goes a step further than previous evaluation challenges by requiring participant systems to perform both event coreference and temporal relation extraction across documents. Four teams submitted the output of their systems to the four proposed subtracks for a total of 13 runs, the best of which obtained an F1-score of 7.85 in the main track (timeline creation from raw text).

[1]  L. R. Dice.  Measures of the Amount of Ecologic Association Between Species , 1945 .

[2]  James Pustejovsky,et al.  TimeML: Robust Specification of Event and Temporal Expressions in Text , 2003, New Directions in Question Answering.

[3]  James Pustejovsky,et al.  The TempEval challenge: identifying temporal relations in text , 2009, Lang. Resour. Evaluation.

[4]  Heng Ji,et al.  Cross-document Event Extraction and Tracking: Task, Evaluation, Techniques and Challenges , 2009, RANLP.

[5]  Yannick Versley,et al.  SemEval-2010 Task 1: Coreference Resolution in Multiple Languages , 2009, *SEMEVAL.

[6]  Olga Babko-Malaya,et al.  PropBank Annotation Guidelines , 2010 .

[7]  Tommaso Caselli,et al.  SemEval-2010 Task 13: TempEval-2 , 2010, *SEMEVAL.

[8]  James F. Allen,et al.  Temporal Evaluation , 2011, ACL.

[9]  James Pustejovsky,et al.  SemEval-2013 Task 1: TempEval-3: Evaluating Time Expressions, Events, and Temporal Relations , 2013, *SEMEVAL.

[10]  Michael Gertz,et al.  HeidelTime: Tuning English and Developing Spanish Resources for TempEval-3 , 2013, *SEMEVAL.

[11]  Steven Bethard,et al.  ClearTK-TimeML: A minimalist approach to TempEval 2013 , 2013, *SEMEVAL.

[12]  Anna Rumshisky,et al.  Evaluating temporal relations in clinical text: 2012 i2b2 Challenge , 2013, J. Am. Medical Informatics Assoc..

[13]  Sara Tonelli,et al.  CROMER: a Tool for Cross-Document Event and Entity Coreference , 2014, LREC.

[14]  Eneko Agirre,et al.  TimeLine: Cross-Document Event Ordering SemEval 2015 - Task 4 , 2015 .