Improving Text Understanding via Deep Syntax-Semantics Communication

Recent studies show that integrating syntactic tree models with sequential semantic models can bring improved task performance, while these methods mostly employ shallow integration of syntax and semantics. In this paper, we propose a deep neural communication model between syntax and semantics to improve the performance of text understanding. Local communication is performed between syntactic tree encoder and sequential semantic encoder for mutual learning of information exchange. Global communication can further ensure comprehensive information propagation. Results on multiple syntax-dependent tasks show that our model outperforms strong baselines by a large margin. In-depth analysis indicates that our method is highly effective in composing sentence semantics.

[1]  Yang Liu,et al.  Structured Alignment Networks for Matching Sentences , 2018, EMNLP.

[2]  Wei Yang,et al.  Incorporating Contextual and Syntactic Structures Improves Semantic Similarity Modeling , 2019, EMNLP/IJCNLP.

[3]  Yufei Wang,et al.  How to best use Syntax in Semantic Role Labelling , 2019, ACL.

[4]  Hermann Ney,et al.  LSTM Neural Networks for Language Modeling , 2012, INTERSPEECH.

[5]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[6]  Marco Marelli,et al.  A SICK cure for the evaluation of compositional distributional semantic models , 2014, LREC.

[7]  Beatrice Santorini,et al.  Building a Large Annotated Corpus of English: The Penn Treebank , 1993, CL.

[8]  Yafeng Ren,et al.  A tree-based neural network model for biomedical event trigger detection , 2020, Inf. Sci..

[9]  Kevin Gimpel,et al.  A Multi-Task Approach for Disentangling Syntax and Semantics in Sentence Representations , 2019, NAACL.

[10]  Christopher Joseph Pal,et al.  Interactive Language Learning by Question Answering , 2019, EMNLP.

[11]  Zhen-Hua Ling,et al.  Enhanced LSTM for Natural Language Inference , 2016, ACL.

[12]  Jian Zhang,et al.  SQuAD: 100,000+ Questions for Machine Comprehension of Text , 2016, EMNLP.

[13]  Isabel Segura-Bedmar,et al.  The 1st DDIExtraction-2011 challenge task: Extraction of Drug-Drug Interactions from biomedical texts , 2011 .

[14]  Shijie Chen,et al.  Technical report on Conversational Question Answering , 2019, ArXiv.

[15]  Xuanjing Huang,et al.  Recurrent Neural Network for Text Classification with Multi-Task Learning , 2016, IJCAI.

[16]  Timothy Dozat,et al.  Deep Biaffine Attention for Neural Dependency Parsing , 2016, ICLR.

[17]  Yue Zhang,et al.  Exploiting Mutual Benefits between Syntax and Semantic Roles using Neural Network , 2016, EMNLP.

[18]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[19]  Richard Johansson,et al.  The CoNLL 2008 Shared Task on Joint Parsing of Syntactic and Semantic Dependencies , 2008, CoNLL.

[20]  Hiroyuki Shindo,et al.  Relation Classification Using Segment-Level Attention-based CNN and Dependency-based RNN , 2019, NAACL-HLT.

[21]  Yue Zhang,et al.  Sentence-State LSTM for Text Representation , 2018, ACL.

[22]  Chiranjib Bhattacharyya,et al.  Incorporating Syntactic and Semantic Information in Word Embeddings using Graph Convolutional Networks , 2018, ACL.

[23]  Christopher D. Manning,et al.  Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks , 2015, ACL.

[24]  Luke S. Zettlemoyer,et al.  Syntactic Scaffolds for Semantic Structures , 2018, EMNLP.

[25]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[26]  David H. Wolpert,et al.  Stacked generalization , 1992, Neural Networks.

[27]  Donghong Ji,et al.  Cross-Lingual Semantic Role Labeling With Model Transfer , 2020, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[28]  Yejin Choi,et al.  Event Detection and Factuality Assessment with Non-Expert Supervision , 2015, EMNLP.

[29]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[30]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[31]  Ido Dagan,et al.  The Third PASCAL Recognizing Textual Entailment Challenge , 2007, ACL-PASCAL@ACL.

[32]  Ido Dagan,et al.  The Sixth PASCAL Recognizing Textual Entailment Challenge , 2009, TAC.

[33]  Christopher Potts,et al.  Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank , 2013, EMNLP.

[34]  Noah A. Smith,et al.  Greedy, Joint Syntactic-Semantic Parsing with Stack LSTMs , 2016, CoNLL.

[35]  Donghong Ji,et al.  Negation and speculation scope detection using recursive neural conditional random fields , 2020, Neurocomputing.

[36]  Armand Joulin,et al.  Cooperative Learning of Disjoint Syntax and Semantics , 2019, NAACL.

[37]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[38]  Peter Norvig,et al.  Deep Learning with Dynamic Computation Graphs , 2017, ICLR.

[39]  Paloma Martínez,et al.  SemEval-2013 Task 9 : Extraction of Drug-Drug Interactions from Biomedical Texts (DDIExtraction 2013) , 2013, *SEMEVAL.

[40]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[41]  Luke S. Zettlemoyer,et al.  Deep Contextualized Word Representations , 2018, NAACL.

[42]  Yue Zhang,et al.  Tree Communication Models for Sentiment Analysis , 2019, ACL.

[43]  Heng Ji,et al.  A Dependency-Based Neural Network for Relation Classification , 2015, ACL.