A Neural Entity Coreference Resolution Review

Abstract Entity Coreference Resolution is the task of resolving all mentions in a document that refer to the same real world entity and is considered as one of the most difficult tasks in natural language understanding. It is of great importance for downstream natural language processing tasks such as entity linking, machine translation, summarization, chatbots, etc. This work aims to give a detailed review of current progress on solving Coreference Resolution using neural-based approaches. It also provides a detailed appraisal of the datasets and evaluation metrics in the field, as well as the subtask of Pronoun Resolution that has seen various improvements in the recent years. We highlight the advantages and disadvantages of the approaches, the challenges of the task, the lack of agreed-upon standards in the task and propose a way to further expand the boundaries of the field.

[1]  Heeyoung Lee,et al.  Stanford’s Multi-Pass Sieve Coreference Resolution System at the CoNLL-2011 Shared Task , 2011, CoNLL Shared Task.

[2]  Breck Baldwin,et al.  Algorithms for Scoring Coreference Chains , 1998 .

[3]  Mitchell P. Marcus,et al.  OntoNotes: The 90% Solution , 2006, NAACL.

[4]  Christopher D. Manning,et al.  Improving Coreference Resolution by Learning Entity-Level Distributed Representations , 2016, ACL.

[5]  R. Mitkov ANAPHORA RESOLUTION: THE STATE OF THE ART , 2007 .

[6]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[7]  Julien Plu,et al.  Knowledge Extraction in Web Media: At The Frontier of NLP, Machine Learning and Semantics , 2016, WWW.

[8]  Alneu de Andrade Lopes,et al.  Word sense disambiguation: A complex network approach , 2018, Inf. Sci..

[9]  Jörg Tiedemann,et al.  ParCor 1.0: A Parallel Pronoun-Coreference Corpus to Support Statistical MT , 2014, LREC.

[10]  Luke S. Zettlemoyer,et al.  Higher-Order Coreference Resolution with Coarse-to-Fine Inference , 2018, NAACL.

[11]  Christopher D. Manning,et al.  Deep Reinforcement Learning for Mention-Ranking Coreference Models , 2016, EMNLP.

[12]  James H. Martin,et al.  Speech and Language Processing, 2nd Edition , 2008 .

[13]  A. Caramazza,et al.  Comprehension of Anaphoric Pronouns. , 1977 .

[14]  Haitao Huang,et al.  Abstractive text summarization using LSTM-CNN based deep learning , 2018, Multimedia Tools and Applications.

[15]  Vincent Ng,et al.  Supervised Noun Phrase Coreference Research: The First Fifteen Years , 2010, ACL.

[16]  Rachel Rudinger,et al.  Gender Bias in Coreference Resolution , 2018, NAACL.

[17]  Christian Hardmeier,et al.  Entity Decisions in Neural Language Modelling: Approaches and Problems , 2019 .

[18]  Dan Roth,et al.  Learning-based Multi-Sieve Co-reference Resolution with Knowledge , 2012, EMNLP-CoNLL.

[19]  Yee Whye Teh,et al.  Distral: Robust multitask reinforcement learning , 2017, NIPS.

[20]  Jonas Kuhn,et al.  Learning Structured Perceptrons for Coreference Resolution with Latent Antecedents and Non-local Features , 2014, ACL.

[21]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[22]  Bo Liu Anonymized BERT: An Augmentation Approach to the Gendered Pronoun Resolution Challenge , 2019, ArXiv.

[23]  Douglas Biber,et al.  Register, Genre, and Style , 2019 .

[24]  Eduard H. Hovy,et al.  BLANC: Implementing the Rand index for coreference evaluation , 2010, Natural Language Engineering.

[25]  Dan Roth,et al.  Understanding the Value of Features for Coreference Resolution , 2008, EMNLP.

[26]  Jackie Chi Kit Cheung,et al.  The Hard-CoRe Coreference Corpus: Removing Gender and Number Cues for Difficult Pronominal Anaphora Resolution , 2018, ArXiv.

[27]  Rui Zhang,et al.  Neural Coreference Resolution with Deep Biaffine Attention by Joint Mention Detection and Mention Clustering , 2018, ACL.

[28]  Omer Levy,et al.  BERT for Coreference Resolution: Baselines and Analysis , 2019, EMNLP/IJCNLP.

[29]  Kamlesh Dutta,et al.  A comprehensive review on feature set used for anaphora resolution , 2020, Artificial Intelligence Review.

[30]  Yejin Choi,et al.  Dynamic Entity Representations in Neural Language Models , 2017, EMNLP.

[31]  Wang Ling,et al.  Reference-Aware Language Models , 2016, EMNLP.

[32]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[33]  Jason Weston,et al.  Learning Anaphoricity and Antecedent Ranking Features for Coreference Resolution , 2015, ACL.

[34]  Xiaoqing Zheng,et al.  Improving Coreference Resolution by Leveraging Entity-Centric Features with Graph Neural Networks and Second-order Inference , 2020, ArXiv.

[35]  Luke S. Zettlemoyer,et al.  Deep Contextualized Word Representations , 2018, NAACL.

[36]  Ido Dagan,et al.  CoRefi: A Crowd Sourcing Suite for Coreference Annotation , 2020, EMNLP.

[37]  Christopher Potts,et al.  The Life and Death of Discourse Entities: Identifying Singleton Mentions , 2013, NAACL.

[38]  Philippe Cudré-Mauroux,et al.  SANAPHOR: Ontology-Based Coreference Resolution , 2015, SEMWEB.

[39]  Jian Su,et al.  An NP-Cluster Based Approach to Coreference Resolution , 2004, COLING.

[40]  Nancy A. Chinchor,et al.  Overview of MUC-7 , 1998, MUC.

[41]  Yan Song,et al.  Knowledge-aware Pronoun Coreference Resolution , 2019, ACL.

[42]  Ander Soraluze,et al.  Deep Cross-Lingual Coreference Resolution for Less-Resourced Languages: The Case of Basque , 2019, Proceedings of the Second Workshop on Computational Models of Reference, Anaphora and Coreference.

[43]  Mark Steedman,et al.  An Annotation Scheme for Information Status in Dialogue , 2004, LREC.

[44]  Michael Strube,et al.  Analyzing and Visualizing Coreference Resolution Errors , 2015, HLT-NAACL.

[45]  Pingjian Zhang,et al.  Entity Candidate Network for Whole-Aware Named Entity Recognition , 2020, ArXiv.

[46]  Jifan Chen,et al.  Multi-hop Question Answering via Reasoning Chains , 2019, ArXiv.

[47]  Rico Sennrich,et al.  Context-Aware Neural Machine Translation Learns Anaphora Resolution , 2018, ACL.

[48]  Michael Strube,et al.  Using Linguistic Features to Improve the Generalization Capability of Neural Coreference Resolvers , 2017, EMNLP.

[49]  Eraldo Rezende Fernandes,et al.  Latent Structure Perceptron with Feature Induction for Unrestricted Coreference Resolution , 2012, EMNLP-CoNLL Shared Task.

[50]  Jiwei Li,et al.  CorefQA: Coreference Resolution as Query-based Span Prediction , 2020, ACL.

[51]  Vincent Ng,et al.  Supervised Models for Coreference Resolution , 2009, EMNLP.

[52]  Vincent Ng,et al.  Unsupervised Models for Coreference Resolution , 2008, EMNLP.

[53]  Hai Zhao,et al.  Lingke: a Fine-grained Multi-turn Chatbot for Customer Service , 2018, COLING.

[54]  Andrei Popescu-Belis,et al.  Context in Neural Machine Translation: A Review of Models and Evaluations , 2019, ArXiv.

[55]  Sopan Khosla,et al.  Using Type Information to Improve Entity Coreference Resolution , 2020, CODI.

[56]  Allyson Ettinger,et al.  Learning to Ignore: Long Document Coreference with Bounded Memory Neural Networks , 2020, EMNLP.

[57]  Ryan Cotterell,et al.  Gender Bias in Contextualized Word Embeddings , 2019, NAACL.

[58]  Yu-Hsin Chen,et al.  Character Identification on Multiparty Conversation: Identifying Mentions of Characters in TV Shows , 2016, SIGDIAL Conference.

[59]  Michael Strube,et al.  Lexical Features in Coreference Resolution: To be Used With Caution , 2017, ACL.

[60]  D. Moher,et al.  Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement , 2009, BMJ : British Medical Journal.

[61]  Michael Strube,et al.  Which Coreference Evaluation Metric Do You Trust? A Proposal for a Link-based Entity Aware Metric , 2016, ACL.

[62]  Hamido Fujita,et al.  Word Sense Disambiguation: A comprehensive knowledge exploitation framework , 2020, Knowl. Based Syst..

[63]  Ch. Satyananda Reddy,et al.  Extractive Text Summarization Using Lexical Association and Graph Based Text Analysis , 2016 .

[64]  H. H. Clark,et al.  What's new? Acquiring New information as a process in comprehension , 1974 .

[65]  Christoph Müller,et al.  Multi-level annotation of linguistic data with MMAX 2 , 2006 .

[66]  John Langford,et al.  Learning to Search for Dependencies , 2015, ArXiv.

[67]  Anders Søgaard,et al.  Model-based annotation of coreference , 2020, LREC.

[68]  Christiane Fellbaum,et al.  A Semantic Network of English: The Mother of All WordNets , 1998, Comput. Humanit..

[69]  Ralph Grishman,et al.  Message Understanding Conference- 6: A Brief History , 1996, COLING.

[70]  Omer Levy,et al.  What Does BERT Look at? An Analysis of BERT’s Attention , 2019, BlackboxNLP@ACL.

[71]  Veselin Stoyanov,et al.  Easy-first Coreference Resolution , 2012, COLING.

[72]  Claire Cardie,et al.  Conundrums in Noun Phrase Coreference Resolution: Making Sense of the State-of-the-Art , 2009, ACL.

[73]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[74]  Lynette Hirschman,et al.  A Model-Theoretic Coreference Scoring Scheme , 1995, MUC.

[75]  Claire Cardie,et al.  Identifying Anaphoric and Non-Anaphoric Noun Phrases to Improve Coreference Resolution , 2002, COLING.

[76]  James R. Curran,et al.  Limited memory incremental coreference resolution , 2014, COLING.

[77]  Dan Klein,et al.  Easy Victories and Uphill Battles in Coreference Resolution , 2013, EMNLP.

[78]  Andrew M. Dai,et al.  Adversarial Training Methods for Semi-Supervised Text Classification , 2016, ICLR.

[79]  Alan W Black,et al.  Measuring Bias in Contextualized Word Representations , 2019, Proceedings of the First Workshop on Gender Bias in Natural Language Processing.

[80]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[81]  Ivan Titov,et al.  Optimizing Differentiable Relaxations of Coreference Evaluation Metrics , 2017, CoNLL.

[82]  Kees van Deemter,et al.  Coreference Annotation: Whither? , 2000, LREC.

[83]  Nathan Schneider,et al.  Semantically Constrained Multilayer Annotation: The Case of Coreference , 2019, Proceedings of the First International Workshop on Designing Meaning Representations.

[84]  Xiaoqiang Luo,et al.  On Coreference Resolution Performance Metrics , 2005, HLT.

[85]  Marilyn Bohl,et al.  Information processing , 1971 .

[86]  Ioannis Vlahavas,et al.  E.T.: Entity-Transformers. Coreference augmented Neural Language Model for richer mention representations via Entity-Transformer blocks , 2020, CRAC.

[87]  Kees van Deemter,et al.  On Coreferring: Coreference in MUC and Related Annotation Schemes , 2000, CL.

[88]  Noam Chomsky,et al.  Lectures on Government and Binding , 1981 .

[89]  Omer Levy,et al.  SpanBERT: Improving Pre-training by Representing and Predicting Spans , 2019, TACL.

[90]  Yan Song,et al.  Incorporating Context and External Knowledge for Pronoun Coreference Resolution , 2019, NAACL.

[91]  Daniel Marcu,et al.  Sentence Level Discourse Parsing using Syntactic and Lexical Information , 2003, NAACL.

[92]  James H. Martin,et al.  Speech and language processing: an introduction to natural language processing, computational linguistics, and speech recognition, 2nd Edition , 2000, Prentice Hall series in artificial intelligence.

[93]  Raphaël Troncy,et al.  Sanaphor++: Combining Deep Neural Networks with Semantics for Coreference Resolution , 2018, LREC.

[94]  Benjamin Van Durme,et al.  Revisiting Memory-Efficient Incremental Coreference Resolution , 2020, ArXiv.

[95]  Gabriel Stanovsky,et al.  Active Learning for Coreference Resolution using Discrete Annotation , 2020, ACL.

[96]  Ali Akbar,et al.  A feature based approach for sentiment analysis using SVM and coreference resolution , 2017, 2017 International Conference on Inventive Communication and Computational Technologies (ICICCT).

[97]  Jason Baldridge,et al.  Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns , 2018, TACL.

[98]  Graeme Hirst,et al.  Anaphora in Natural Language Understanding: A Survey , 1981, Lecture Notes in Computer Science.

[99]  Fethiye Irmak Dogan,et al.  Fantom: A Crowdsourced Social Chatbot using an Evolving Dialog Graph , 2018 .

[100]  Hwee Tou Ng,et al.  A Machine Learning Approach to Coreference Resolution of Noun Phrases , 2001, CL.

[101]  Dan Roth,et al.  Evaluation of named entity coreference , 2019, Proceedings of the Second Workshop on Computational Models of Reference, Anaphora and Coreference.

[102]  B. Byrne,et al.  Neural Machine Translation Doesn’t Translate Gender Coreference Right Unless You Make It , 2020, GEBNLP.

[103]  Anna Rumshisky,et al.  Triad-based Neural Network for Coreference Resolution , 2018, COLING.

[104]  Christian Hardmeier,et al.  ParCorFull: a Parallel Corpus Annotated with Full Coreference , 2018, LREC.

[105]  Jian Su,et al.  Coreference Resolution Using Competition Learning Approach , 2003, ACL.

[106]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[107]  Daniel S. Weld,et al.  Design Challenges for Entity Linking , 2015, TACL.

[108]  James R. Glass,et al.  Learning Word Representations with Cross-Sentence Dependency for End-to-End Co-reference Resolution , 2018, EMNLP.

[109]  Yang Trista Cao,et al.  Toward Gender-Inclusive Coreference Resolution , 2019, ACL.

[110]  Philippe Langlais,et al.  WikiCoref: An English Coreference-annotated Corpus of Wikipedia Articles , 2016, LREC.

[111]  Jieyu Zhao,et al.  Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods , 2018, NAACL.

[112]  Nianwen Xue,et al.  CoNLL-2011 Shared Task: Modeling Unrestricted Coreference in OntoNotes , 2011, CoNLL Shared Task.

[113]  Michael Strube,et al.  Evaluation Metrics For End-to-End Coreference Resolution Systems , 2010, SIGDIAL Conference.

[114]  Amir Zeldes,et al.  ANNIS3: A new architecture for generic corpus query and visualization , 2016, Digit. Scholarsh. Humanit..

[115]  Pascal Denis,et al.  Joint Determination of Anaphoricity and Coreference Resolution using Integer Programming , 2007, NAACL.

[116]  David Bamman,et al.  An Annotated Dataset of Coreference in English Literature , 2020, LREC.

[117]  Jackie Chi Kit Cheung,et al.  The KnowRef Coreference Corpus: Removing Gender and Number Cues for Difficult Pronominal Anaphora Resolution , 2018, ACL.

[118]  Luciano da Fontoura Costa,et al.  Using complex networks for text classification: Discriminating informative and imaginative documents , 2016 .

[119]  Michel Gagnon,et al.  Poly-co: a multilayer perceptron approach for coreference detection , 2011, CoNLL Shared Task.

[120]  Andrew M. Dai,et al.  Virtual Adversarial Training for Semi-Supervised Text Classification , 2016, ArXiv.

[121]  Chen Qiu,et al.  Rewarding Coreference Resolvers for Being Consistent with World Knowledge , 2019, EMNLP/IJCNLP.

[122]  Xiaoqiang Luo,et al.  Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation , 2014, ACL.

[123]  Amir Globerson,et al.  Coreference Resolution with Entity Equalization , 2019, ACL.

[124]  Thanh Hung Vo,et al.  Sentiment Analysis Using Anaphoric Coreference Resolution and Ontology Inference , 2016, MIWAI.

[125]  Mari Ostendorf,et al.  Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction , 2018, EMNLP.

[126]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[127]  Mark A. Przybocki,et al.  The Automatic Content Extraction (ACE) Program – Tasks, Data, and Evaluation , 2004, LREC.

[128]  Alexander M. Rush,et al.  Learning Global Features for Coreference Resolution , 2016, NAACL.

[129]  Liyan Xu,et al.  Revealing the Myth of Higher-Order Inference in Coreference Resolution , 2020, EMNLP.

[130]  Hong Chen,et al.  PreCo: A Large-scale Dataset in Preschool Vocabulary for Coreference Resolution , 2018, EMNLP.

[131]  Ellen F. Prince,et al.  Toward a taxonomy of given-new information , 1981 .

[132]  Shalom Lappin,et al.  An Algorithm for Pronominal Anaphora Resolution , 1994, CL.

[133]  Erik Cambria,et al.  Anaphora and Coreference Resolution: A Review , 2018, Inf. Fusion.

[134]  Zhen-Hua Ling,et al.  A Study on Improving End-to-End Neural Coreference Resolution , 2018, CCL.

[135]  Alex Wang,et al.  What do you learn from context? Probing for sentence structure in contextualized word representations , 2019, ICLR.

[136]  Dan Roth,et al.  A Constrained Latent Variable Model for Coreference Resolution , 2013, EMNLP.

[137]  Amir Zeldes,et al.  The GUM corpus: creating multilayer resources in the classroom , 2016, Language Resources and Evaluation.

[138]  Xiaoqiang Luo,et al.  A Mention-Synchronous Coreference Resolution Algorithm Based On the Bell Tree , 2004, ACL.

[139]  Elena Lloret,et al.  NATSUM: Narrative abstractive summarization through cross-document timeline generation , 2019, Inf. Process. Manag..

[140]  Dan Roth,et al.  Improving Generalization in Coreference Resolution via Adversarial Training , 2019, *SEMEVAL.

[141]  Chen Chen,et al.  Linguistically Aware Coreference Evaluation Metrics , 2013, IJCNLP.

[142]  Hongliang Fei,et al.  Coreference Aware Representation Learning for Neural Named Entity Recognition , 2019, IJCAI.

[143]  Sandeep Attree,et al.  Gendered Ambiguous Pronouns Shared Task: Boosting Model Confidence by Evidence Pooling , 2019, Proceedings of the First Workshop on Gender Bias in Natural Language Processing.

[144]  Luke S. Zettlemoyer,et al.  End-to-end Neural Coreference Resolution , 2017, EMNLP.

[145]  Steven Skiena,et al.  Polyglot: Distributed Word Representations for Multilingual NLP , 2013, CoNLL.

[146]  Christopher D. Manning,et al.  Entity-Centric Coreference Resolution with Model Stacking , 2015, ACL.

[147]  H. Kuhn The Hungarian method for the assignment problem , 1955 .

[148]  Gourab Kundu,et al.  Neural Cross-Lingual Coreference Resolution And Its Application To Entity Linking , 2018, ACL.

[149]  Junlin Yang,et al.  Look Again at the Syntax: Relational Graph Convolutional Network for Gendered Ambiguous Pronoun Resolution , 2019, Proceedings of the First Workshop on Gender Bias in Natural Language Processing.

[150]  Yuchen Zhang,et al.  CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes , 2012, EMNLP-CoNLL Shared Task.

[151]  Rejwanul Haque,et al.  Investigating Query Expansion and Coreference Resolution in Question Answering on BERT , 2020, NLDB.

[152]  Luke S. Zettlemoyer,et al.  Syntactic Scaffolds for Semantic Structures , 2018, EMNLP.

[153]  Eva Hajicová,et al.  The Role of the Hierarchy of Activation in the Process of Natural Language Understanding , 1982, COLING.