Error-Driven Analysis of Challenges in Coreference Resolution

Coreference resolution metrics quantify errors but do not analyze them. Here, we consider an automated method of categorizing errors in the output of a coreference system into intuitive underlying error types. Using this tool, we first compare the error distributions across a large set of systems, then analyze common errors across the top ten systems, empirically characterizing the major unsolved challenges of the coreference resolution task.

[1]  Bernard Callinan.  Sir John Monash , 1981 .

[2]  C. D. Paice,et al.  Towards the automatic recognition of anaphoric features in English text: the impersonal pronoun “it” , 1987 .

[3]  Lynette Hirschman,et al.  A Model-Theoretic Coreference Scoring Scheme , 1995, MUC.

[4]  Breck Baldwin,et al.  Algorithms for Scoring Coreference Chains , 1998 .

[5]  Michael Comstock.  경험 규칙에 의한 대명사의 Coreference Resolution , 2001 .

[6]  Claire Gardent,et al.  Improving Machine Learning Approaches to Coreference Resolution , 2002, ACL.

[7]  Xiaoqiang Luo,et al.  On Coreference Resolution Performance Metrics , 2005, HLT.

[8]  Mitchell P. Marcus,et al.  OntoNotes: The 90% Solution , 2006, NAACL.

[9]  Ralph Weischedel,et al.  Unrestricted Coreference: Identifying Entities and Events in OntoNotes , 2007 .

[10]  Sameer Pradhan,et al.  Unrestricted Coreference: Identifying Entities and Events in OntoNotes , 2007, International Conference on Semantic Computing (ICSC 2007).

[11]  Dan Roth,et al.  Understanding the Value of Features for Coreference Resolution , 2008, EMNLP.

[12]  Vincent Ng,et al.  Unsupervised Models for Coreference Resolution , 2008, EMNLP.

[13]  Yannick Versley,et al.  BART: A Modular Toolkit for Coreference Resolution , 2008, ACL.

[14]  Claire Cardie,et al.  Conundrums in Noun Phrase Coreference Resolution: Making Sense of the State-of-the-Art , 2009, ACL.

[15]  Dan Klein,et al.  Simple Coreference Resolution with Rich Syntactic and Semantic Features , 2009, EMNLP.

[16]  Vincent Ng,et al.  Supervised Models for Coreference Resolution , 2009, EMNLP.

[17]  Claire Cardie,et al.  Coreference Resolution with Reconcile , 2010, ACL.

[18]  M. R E C A S E,et al.  BLANC: Implementing the Rand index for coreference evaluation , 2010, Natural Language Engineering.

[19]  Eduard H. Hovy,et al.  BLANC: Implementing the Rand index for coreference evaluation , 2010, Natural Language Engineering.

[20]  Yang Liu,et al.  ETS: An Error Tolerable System for Coreference Resolution , 2011, CoNLL Shared Task.

[21]  Sameer Pradhan.  Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, CoNLL 2011, Portland, Oregon, USA, June 23-24, 2011 , 2011, CoNLL Shared Task.

[22]  Sobha Lalitha Devi,et al.  Hybrid Approach for Coreference Resolution , 2011, CoNLL Shared Task.

[23]  Xuan Wang,et al.  Coreference Resolution with Loose Transitivity Constraints , 2011, CoNLL Shared Task.

[24]  Nianwen Xue,et al.  A Machine Learning-Based Coreference Detection System for OntoNotes , 2011, CoNLL Shared Task.

[25]  Claire Cardie,et al.  Reconciling OntoNotes: Unrestricted Coreference Resolution in OntoNotes with Reconcile , 2011, CoNLL Shared Task.

[26]  Sandra Kübler,et al.  UBIU: A Robust System for Resolving Unrestricted Coreference , 2011, CoNLL Shared Task.

[27]  Lluís Padró,et al.  RelaxCor Participation in CoNLL Shared Task on Coreference Resolution , 2011, CoNLL Shared Task.

[28]  Dan Roth,et al.  Inference Protocols for Coreference Resolution , 2011, CoNLL Shared Task.

[29]  Wei-Peng Chen,et al.  Coreference Resolution System using Maximum Entropy Classifier , 2011, CoNLL Shared Task.

[30]  Heeyoung Lee,et al.  Stanford’s Multi-Pass Sieve Coreference Resolution System at the CoNLL-2011 Shared Task , 2011, CoNLL Shared Task.

[31]  Michel Gagnon,et al.  Poly-co: a multilayer perceptron approach for coreference detection , 2011, CoNLL Shared Task.

[32]  Nianwen Xue,et al.  CoNLL-2011 Shared Task: Modeling Unrestricted Coreference in OntoNotes , 2011, CoNLL Shared Task.

[33]  Yang Song,et al.  Link Type Based Pre-Cluster Pair Model for Coreference Resolution , 2011, CoNLL Shared Task.

[34]  Cícero Nogueira dos Santos,et al.  Rule and Tree Ensembles for Unrestricted Coreference Resolution , 2011, CoNLL Shared Task.

[35]  Yuji Matsumoto,et al.  Narrative Schema as World Knowledge for Coreference Resolution , 2011, CoNLL Shared Task.

[36]  Yan Zhang,et al.  Combining Syntactic and Semantic Features by SVM for Unrestricted Coreference Resolution , 2011, CoNLL Shared Task.

[37]  Michael Strube,et al.  Unrestricted Coreference Resolution via Global Hypergraph Partitioning , 2011, CoNLL Shared Task.

[38]  Hinrich Schütze,et al.  Supervised Coreference Resolution with SUCRE , 2011, CoNLL Shared Task.

[39]  Pierre Nugues,et al.  Exploring Lexicalized Features for Coreference Resolution , 2011, CoNLL Shared Task.

[40]  Asif Ekbal,et al.  Multi-metric optimization for coreference: The UniTN / IITP / Essex submission to the 2011 CONLL Shared Task , 2011, CoNLL Shared Task.

[41]  Don Tuggener,et al.  An Incremental Model for Coreference Resolution with Restrictive Antecedent Accessibility , 2011, CoNLL Shared Task.

[42]  Dan Klein,et al.  Mention Detection: Heuristics for the OntoNotes annotations , 2011, CoNLL Shared Task.

[43]  Yu-Yin Hsu,et al.  UBIU for Multilingual Coreference Resolution in OntoNotes , 2012, EMNLP-CoNLL Shared Task.

[44]  Dan Klein,et al.  Parser Showdown at the Wall Street Corral: An Empirical Investigation of Error Types in Parser Output , 2012, EMNLP.

[45]  Chen Chen,et al.  Combining the Best of Two Worlds: A Hybrid Approach to Multilingual Coreference Resolution , 2012, EMNLP-CoNLL Shared Task.

[46]  Yang Xiang,et al.  A Mixed Deterministic Model for Coreference Resolution , 2012, EMNLP-CoNLL Shared Task.

[47]  Richárd Farkas,et al.  Data-driven Multilingual Coreference Resolution using Resolver Stacking , 2012, EMNLP-CoNLL Shared Task.

[48]  Yuchen Zhang,et al.  CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes , 2012, EMNLP-CoNLL Shared Task.

[49]  Michael Strube,et al.  A Multigraph Model for Coreference Resolution , 2012, EMNLP-CoNLL Shared Task.

[50]  Gordana Ilic Holen.  Critical Reflections on Evaluation Practices in Coreference Resolution , 2013, NAACL.

[51]  Dan Klein,et al.  Easy Victories and Uphill Battles in Coreference Resolution , 2013, EMNLP.

[52]  Heeyoung Lee,et al.  Deterministic Coreference Resolution Based on Entity-Centric, Precision-Ranked Rules , 2013, CL.