Deep Reinforcement Learning for Mention-Ranking Coreference Models

Coreference resolution systems are typically trained with heuristic loss functions that require careful tuning. In this paper we instead apply reinforcement learning to directly optimize a neural mention-ranking model for coreference evaluation metrics. We experiment with two approaches: the REINFORCE policy gradient algorithm and a reward-rescaled max-margin objective. We find the latter to be more effective, resulting in significant improvements over the current state-of-the-art on the English and Chinese portions of the CoNLL 2012 Shared Task.

[1]  Lynette Hirschman,et al.  A Model-Theoretic Coreference Scoring Scheme , 1995, MUC.

[2]  Breck Baldwin,et al.  Algorithms for Scoring Coreference Chains , 1998 .

[3]  Very Large Corpora.  Empirical Methods in Natural Language Processing , 1999 .

[4]  Ronald J. Williams,et al.  Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning , 2004, Machine Learning.

[5]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[6]  Xiaoqiang Luo,et al.  On Coreference Resolution Performance Metrics , 2005, HLT.

[7]  Daniel Marcu,et al.  Practical structured learning techniques for natural language processing , 2006 .

[8]  Pascal Denis,et al.  A Ranking Approach to Pronoun Resolution , 2007, IJCAI.

[9]  Vincent Ng,et al.  Unsupervised Models for Coreference Resolution , 2008, EMNLP.

[10]  John Langford,et al.  Search-based structured prediction , 2009, Machine Learning.

[11]  Vincent Ng,et al.  Supervised Models for Coreference Resolution , 2009, EMNLP.

[12]  Heeyoung Lee,et al.  A Multi-Pass Sieve for Coreference Resolution , 2010, EMNLP.

[13]  Dan Klein,et al.  Coreference Resolution in a Modular, Entity-Centered Model , 2010, NAACL.

[14]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[15]  Nitish Srivastava,et al.  Improving neural networks by preventing co-adaptation of feature detectors , 2012, ArXiv.

[16]  Eraldo Rezende Fernandes,et al.  Latent Structure Perceptron with Feature Induction for Unrestricted Coreference Resolution , 2012, EMNLP-CoNLL Shared Task.

[17]  Oren Etzioni,et al.  RevMiner: an extractive interface for navigating reviews on a smartphone , 2012, UIST.

[18]  Yuchen Zhang,et al.  CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes , 2012, EMNLP-CoNLL Shared Task.

[19]  Dan Klein,et al.  Easy Victories and Uphill Battles in Coreference Resolution , 2013, EMNLP.

[20]  Dan Klein,et al.  Decentralized Entity-Level Modeling for Coreference Resolution , 2013, ACL.

[21]  Thomas G. Dietterich,et al.  Prune-and-Score: Learning for Greedy Coreference Resolution , 2014, EMNLP.

[22]  Jonas Kuhn,et al.  Learning Structured Perceptrons for Coreference Resolution with Latent Antecedents and Non-local Features , 2014, ACL.

[23]  Michael Strube,et al.  Latent Structures for Coreference Resolution , 2015, TACL.

[24]  Jason Weston,et al.  Learning Anaphoricity and Antecedent Ranking Features for Coreference Resolution , 2015, ACL.

[25]  Christopher D. Manning,et al.  Entity-Centric Coreference Resolution with Model Stacking , 2015, ACL.

[26]  Alexander M. Rush,et al.  Learning Global Features for Coreference Resolution , 2016, NAACL.

[27]  Jure Leskovec,et al.  Large-scale Analysis of Counseling Conversations: An Application of Natural Language Processing to Mental Health , 2016, TACL.

[28]  Jure Leskovec,et al.  Inducing Domain-Specific Sentiment Lexicons from Unlabeled Corpora , 2016, EMNLP.

[29]  Christopher D. Manning,et al.  Improving Coreference Resolution by Learning Entity-Level Distributed Representations , 2016, ACL.

[30]  Elizabeth D. Mynatt,et al.  User Interface Software And Technology , 2017 .