CoNLL-2011 Shared Task: Modeling Unrestricted Coreference in OntoNotes

The CoNLL-2011 shared task involved predicting coreference using OntoNotes data. Resources in this field have tended to be limited to noun phrase coreference, often on a restricted set of entities, such as ace entities. OntoNotes provides a large-scale corpus of general anaphoric coreference not restricted to noun phrases or to a specified set of entity types. OntoNotes also provides additional layers of integrated annotation, capturing additional shallow semantic structure. This paper briefly describes the OntoNotes annotation (coreference and other layers) and then describes the parameters of the shared task including the format, pre-processing information, and evaluation criteria, and presents and discusses the results achieved by the participating systems. Having a standard test set and evaluation parameters, all based on a new resource that provides multiple integrated annotation layers (parses, semantic roles, word senses, named entities and coreference) that could support joint models, should help to energize ongoing research in the task of entity and event coreference.

[1]  Neville Ryant,et al.  A Large-scale Classication of English Verbs , 2006 .

[2]  M. R E C A S E,et al.  BLANC: Implementing the Rand index for coreference evaluation , 2010 .

[3]  Richard Johansson,et al.  The CoNLL 2008 Shared Task on Joint Parsing of Syntactic and Semantic Dependencies , 2008, CoNLL.

[4]  Andrew McCallum,et al.  Conditional Models of Identity Uncertainty with Application to Noun Coreference , 2004, NIPS.

[5]  Richard Johansson,et al.  The CoNLL-2009 Shared Task: Syntactic and Semantic Dependencies in Multiple Languages , 2009, CoNLL Shared Task.

[6]  Simone Paolo Ponzetto,et al.  Exploiting Semantic Role Labeling, WordNet and Wikipedia for Coreference Resolution , 2006, NAACL.

[7]  Vincent Ng,et al.  Supervised Models for Coreference Resolution , 2009, EMNLP.

[8]  Vincent Ng,et al.  Supervised Noun Phrase Coreference Research: The First Fifteen Years , 2010, ACL.

[9]  Sanda M. Harabagiu,et al.  RESOLUTION , 1977, Monatsschrift für Kriminologie und Strafrechtsreform.

[10]  Mitchell P. Marcus,et al.  OntoNotes: The 90% Solution , 2006, NAACL.

[11]  Dekang Lin,et al.  Bootstrapping Path-Based Pronoun Resolution , 2006, ACL.

[12]  Massimo Poesio,et al.  The MATE/GNOME Proposals for Anaphoric Annotation, Revisited , 2004, SIGDIAL Workshop.

[13]  Dan Klein,et al.  Coreference Resolution in a Modular, Entity-Centered Model , 2010, NAACL.

[14]  Claire Cardie,et al.  Conundrums in Noun Phrase Coreference Resolution: Making Sense of the State-of-the-Art , 2009, ACL.

[15]  Yannick Versley,et al.  SemEval-2010 Task 1: Coreference Resolution in Multiple Languages , 2009, *SEMEVAL.

[16]  Breck Baldwin,et al.  Algorithms for Scoring Coreference Chains , 1998 .

[17]  Mitchell P. Marcus,et al.  OntoNotes: A Unified Relational Semantic Representation , 2007, International Conference on Semantic Computing (ICSC 2007).

[18]  Massimo Poesio,et al.  State-of-the-art NLP Approaches to Coreference Resolution: Theory and Practical Recipes , 2009, ACL.

[19]  Ron Artstein,et al.  The Reliability of Anaphoric Annotation, Reconsidered: Taking Ambiguity into Account , 2005, FCA@ACL.

[20]  Mark A. Przybocki,et al.  The Automatic Content Extraction (ACE) Program – Tasks, Data, and Evaluation , 2004, LREC.

[21]  Hwee Tou Ng,et al.  A Machine Learning Approach to Coreference Resolution of Noun Phrases , 2001, CL.

[22]  Rebecca J. Passonneau.  Computing Reliability for Coreference Annotation , 2004, LREC.

[23]  Sameer Pradhan,et al.  Unrestricted Coreference: Identifying Entities and Events in OntoNotes , 2007, International Conference on Semantic Computing (ICSC 2007).

[24]  Eugene Charniak,et al.  Coarse-to-Fine n-Best Parsing and MaxEnt Discriminative Reranking , 2005, ACL.

[25]  Daniel Gildea,et al.  The Proposition Bank: An Annotated Corpus of Semantic Roles , 2005, CL.

[26]  Christopher R. Johnson,et al.  Background to Framenet , 2003 .

[27]  Yannick Versley.  Antecedent Selection Techniques for High-Recall Coreference Resolution , 2007, EMNLP-CoNLL.

[28]  Wendy G. Lehnert,et al.  Using Decision Trees for Coreference Resolution , 1995, IJCAI.

[29]  Neville Ryant,et al.  A large-scale classification of English verbs , 2008, Lang. Resour. Evaluation.

[30]  Daniel Jurafsky,et al.  Support Vector Learning for Semantic Argument Classification , 2005, Machine Learning.

[31]  Andrew McCallum,et al.  First-Order Probabilistic Models for Coreference Resolution , 2007, NAACL.

[32]  Thomas S. Morton,et al.  Coreference for NLP Applications , 2000, ACL.

[33]  Simone Paolo Ponzetto,et al.  Semantic Role Labeling for Coreference Resolution , 2006, EACL.

[34]  Vincent Ng,et al.  Unsupervised Models for Coreference Resolution , 2008, EMNLP.

[35]  Lynette Hirschman,et al.  A Model-Theoretic Coreference Scoring Scheme , 1995, MUC.

[36]  Eduard H. Hovy,et al.  BLANC: Implementing the Rand index for coreference evaluation , 2010, Natural Language Engineering.

[37]  Pascal Denis,et al.  Joint Determination of Anaphoricity and Coreference Resolution using Integer Programming , 2007, NAACL.

[38]  Michael Strube,et al.  Evaluation Metrics For End-to-End Coreference Resolution Systems , 2010, SIGDIAL Conference.

[39]  Beatrice Santorini,et al.  Building a Large Annotated Corpus of English: The Penn Treebank , 1993, CL.

[40]  Ralph Weischedel,et al.  Unrestricted Coreference: Identifying Entities and Events in OntoNotes , 2007 .

[41]  William M. Rand,et al.  Objective Criteria for the Evaluation of Clustering Methods , 1971 .

[42]  Vincent Ng,et al.  Shallow Semantics for Coreference Resolution , 2007, IJCAI.

[43]  Seth Kulick,et al.  Issues in Synchronizing the English Treebank and PropBank , 2006 .

[44]  Philipp Koehn,et al.  Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL) , 2007 .

[45]  Eugene Charniak,et al.  Edit Detection and Parsing for Transcribed Speech , 2001, NAACL.

[46]  Xiaoqiang Luo,et al.  On Coreference Resolution Performance Metrics , 2005, HLT.

[47]  Pascal Denis,et al.  Global joint models for coreference resolution and named entity classification , 2009, Proces. del Leng. Natural.

[48]  Christiane Fellbaum,et al.  Making fine-grained and coarse-grained sense distinctions, both manually and automatically , 2006, Natural Language Engineering.