ERD'14

In this paper we overview the 2014 Entity Recognition and Disambiguation Challenge (ERD'14), which took place from March to June 2014 and was summarized in a dedicated workshop at SIGIR 2014. The main goal of the ERD challenge was to promote research in recognition and disambiguation of entities in unstructured text. Unlike many past entity linking challenges, no mention segmentations were given to the participating systems for a given document. Participants were asked to implement a web service for their system to minimize human involvement during evaluation and to enable measuring the processing times. The challenge has attracted a lot of interest (over 100 teams registered, and 27 of those submitted final results). In this paper we cover the task definition, issues encountered during annotation, and provide a detailed analysis of all the participating systems. Specifically, we show how we adapted the pooling technique to address the difficulties of gathering annotations for the entity linking task. We also summarize the ERD workshop that followed the challenge, including the oral and poster presentations as well as the invited talks.

[1]  Vasudeva Varma,et al.  Exploiting Wikipedia inlinks for linking entities in queries , 2014, ERD '14.

[2]  Krisztian Balog,et al.  A greedy algorithm for finding sets of entity linking interpretations in queries , 2014, ERD '14.

[3]  Eneko Agirre,et al.  UBC entity recognition and disambiguation at ERD 2014 , 2014, ERD '14.

[4]  Pararth Shah,et al.  System for collective entity disambiguation , 2014, ERD '14.

[5]  Chuan Wu,et al.  An optimization framework for entity recognition and disambiguation , 2014, ERD '14.

[6]  Hsin-Hsi Chen,et al.  NTUNLP Approaches to Recognizing and Disambiguating Entities in Long and Short Text in the 2014 ERD Challenge , 2014 .

[7]  Heng Ji,et al.  Overview of the TAC 2010 Knowledge Base Population Track , 2010 .

[8]  Sam Steingold,et al.  A search based approach to entity recognition: magnetic and IISAS team at ERD challenge , 2014, ERD '14.

[9]  Maarten Marx,et al.  Entity linking by focusing DBpedia candidate entities , 2014, ERD '14.

[10]  Evangelos E. Milios,et al.  Tulip: lightweight entity recognition and disambiguation using wikipedia-based topic centroids , 2014, ERD '14.

[11]  Peter Adolphs,et al.  The neofonie NERD system at the ERD challenge 2014 , 2014, ERD '14.

[12]  Pablo N. Mendes,et al.  Improving efficiency and accuracy in multilingual entity extraction , 2013, I-SEMANTICS '13.

[13]  C. J. van Rijsbergen,et al.  Report on the need for and provision of an 'ideal' information retrieval test collection , 1975 .

[14]  Maarten de Rijke,et al.  Semanticizing search engine queries: the University of Amsterdam at the ERD 2014 challenge , 2014, ERD '14.

[15]  Ellen M. Voorhees,et al.  Overview of TREC 2001 , 2001, TREC.

[16]  Justin Zobel,et al.  How reliable are the results of large-scale information retrieval experiments? , 1998, SIGIR '98.

[17]  James P. Callan,et al.  A language modeling approach to entity recognition and disambiguation for search queries , 2014, ERD '14.

[18]  Paolo Ferragina,et al.  TAGME: on-the-fly annotation of short text fragments (by wikipedia entities) , 2010, CIKM.

[19]  Hinrich Schütze,et al.  The SMAPH system for query entity recognition and disambiguation , 2014, ERD '14.

[20]  Paolo Ferragina,et al.  From TagME to WAT: a new entity annotator , 2014, ERD '14.

[21]  Horacio Rodríguez,et al.  The TALP participation at ERD 2014 , 2014, ERD '14.