Second-Order Semantic Role Labeling With Global Structural Refinement

Semantic role labeling (SRL) is one of the important tasks in natural language processing. Current end-to-end SRL, compared with traditional pipeline SRL, has achieved competitive performance via graph-based neural models. However, these are all first-order methods, where decisions for detecting predicate-argument pairs are made in isolation with local short-term features, thus being error-prone. Besides, they may potentially suffer from long-distance dependency, and perform poorly on the data-scarce scenario. In this paper, we explore a second-order end-to-end SRL model that considers simultaneously two pairs of predicate-argument when making scoring. We next propose a structural refinement mechanism to further model higher-order interactions at a global scope. After iterations of refinement, the underlying token representations are refined to be more informative. Experimental results show that our model significantly outperforms the state-of-the-art baselines on multiple benchmarks (i.e., CoNLL08, CoNLL09 and CoNLL12). Further analysis demonstrates that our framework is able to learn latent SRL constraints, being more robust on data-scarce settings, meanwhile relieving the long-range dependency issue.

[1]  Hiroyuki Shindo,et al.  A Span Selection Model for Semantic Role Labeling , 2018, EMNLP.

[2]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[3]  Richard Johansson,et al.  The CoNLL-2009 Shared Task: Syntactic and Semantic Dependencies in Multiple Languages , 2009, CoNLL Shared Task.

[4]  Richard Johansson,et al.  The CoNLL 2008 Shared Task on Joint Parsing of Syntactic and Semantic Dependencies , 2008, CoNLL.

[5]  Pierre Nugues,et al.  A High-Performance Syntactic and Semantic Dependency Parser , 2010, COLING.

[6]  Luke S. Zettlemoyer,et al.  Deep Contextualized Word Representations , 2018, NAACL.

[7]  Hwee Tou Ng,et al.  Towards Robust Linguistic Analysis using OntoNotes , 2013, CoNLL.

[8]  Donghong Ji,et al.  Boundaries and edges rethinking: An end-to-end neural model for overlapping entity relation extraction , 2020, Inf. Process. Manag..

[9]  Luo Si,et al.  Syntax-Enhanced Self-Attention-Based Semantic Role Labeling , 2019, EMNLP/IJCNLP.

[10]  Hai Zhao,et al.  Fourth-Order Dependency Parsing , 2012, COLING.

[11]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[12]  Donghong Ji,et al.  Cross-Lingual Semantic Role Labeling With Model Transfer , 2020, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[13]  Omer Levy,et al.  Jointly Predicting Predicates and Arguments in Neural Semantic Role Labeling , 2018, ACL.

[14]  Donghong Ji,et al.  Cross-Lingual Semantic Role Labeling with High-Quality Translated Training Corpus , 2020, ACL.

[15]  Christian Scheible,et al.  An Evaluation of Predicate Argument Clustering using Pseudo-Disambiguation , 2010, LREC.

[16]  Vivek Srikumar,et al.  Structured Tuning for Semantic Role Labeling , 2020, ACL.

[17]  Donghong Ji,et al.  End-to-end Semantic Role Labeling with Neural Transition-based Model , 2021, AAAI.

[18]  Hai Zhao,et al.  Semantic Dependency Parsing of NomBank and PropBank: An Efficient Integrated Approach via a Large-scale Feature Selection , 2009, EMNLP.

[19]  Jimmy J. Lin,et al.  Simple BERT Models for Relation Extraction and Semantic Role Labeling , 2019, ArXiv.

[20]  Koby Crammer,et al.  Online Large-Margin Training of Dependency Parsers , 2005, ACL.

[21]  Daniel Jurafsky,et al.  Semantic Role Labeling Using Different Syntactic Views , 2005, ACL.

[22]  Kuzman Ganchev,et al.  Semantic Role Labeling with Neural Network Factors , 2015, EMNLP.

[23]  Timothy Dozat,et al.  Deep Biaffine Attention for Neural Dependency Parsing , 2016, ICLR.

[24]  Min Zhang,et al.  A Syntax-aware Multi-task Learning Framework for Chinese Semantic Role Labeling , 2019, EMNLP.

[25]  Xavier Carreras,et al.  Experiments with a Higher-Order Projective Dependency Parser , 2007, EMNLP.

[26]  Donghong Ji,et al.  High-order Refining for End-to-end Chinese Semantic Role Labeling , 2020, AACL/IJCNLP.

[27]  Diego Marcheggiani,et al.  Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling , 2017, EMNLP.

[28]  Diego Marcheggiani,et al.  A Simple and Accurate Syntax-Agnostic Neural Model for Dependency-based Semantic Role Labeling , 2017, CoNLL.

[29]  Hai Zhao,et al.  A Full End-to-End Semantic Role Labeler, Syntactic-agnostic Over Syntactic-aware? , 2018, COLING.

[30]  Dong-Hong Ji,et al.  Encoder-Decoder Based Unified Semantic Role Labeling with Label-Aware Syntax , 2021, AAAI.

[31]  Dan Roth,et al.  Semantic Role Labeling Via Generalized Inference Over Classifiers , 2004, CoNLL.

[32]  Luo Si,et al.  Syntax-Aware Neural Semantic Role Labeling , 2019, AAAI.

[33]  Michael Collins,et al.  Efficient Third-Order Dependency Parsers , 2010, ACL.

[34]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[35]  Mirella Lapata,et al.  Semi-Supervised Semantic Role Labeling with Cross-View Training , 2019, EMNLP.

[36]  Hai Zhao,et al.  High-order Semantic Role Labeling , 2020, FINDINGS.

[37]  Kewei Tu,et al.  Second-Order Semantic Dependency Parsing with End-to-End Neural Networks , 2019, ACL.

[38]  Shay B. Cohen,et al.  Semantic Role Labeling with Iterative Structure Refinement , 2019, EMNLP.

[39]  Luke S. Zettlemoyer,et al.  Deep Semantic Role Labeling: What Works and What’s Next , 2017, ACL.

[40]  Hai Zhao,et al.  Syntax for Semantic Role Labeling, To Be, Or Not To Be , 2018, ACL.

[41]  Andrew McCallum,et al.  Linguistically-Informed Self-Attention for Semantic Role Labeling , 2018, EMNLP.

[42]  Jason Eisner,et al.  Three New Probabilistic Models for Dependency Parsing: An Exploration , 1996, COLING.

[43]  Dan Roth,et al.  The Importance of Syntactic Parsing and Inference in Semantic Role Labeling , 2008, CL.

[44]  Mirella Lapata,et al.  Neural Semantic Role Labeling with Dependency Path Embeddings , 2016, ACL.

[45]  Hai Zhao,et al.  Dependency or Span, End-to-End Uniform Semantic Role Labeling , 2019, AAAI.

[46]  Alessandro Moschitti,et al.  High-Order Low-Rank Tensors for Semantic Role Labeling , 2015, HLT-NAACL.

[47]  Daniel Gildea,et al.  Automatic Labeling of Semantic Roles , 2000, ACL.

[48]  Xinchi Chen,et al.  Capturing Argument Interaction in Semantic Role Labeling with Capsule Networks , 2019, EMNLP.

[49]  Hai Zhao,et al.  Multilingual Dependency Learning: Exploiting Rich Features for Tagging Syntactic and Semantic Dependencies , 2009, CoNLL Shared Task.

[50]  Pierre Nugues,et al.  Multilingual Semantic Role Labeling , 2009, CoNLL Shared Task.