Tree Framework With BERT Word Embedding for the Recognition of Chinese Implicit Discourse Relations

Currently, discourse relation recognition (DRR), which is not directly marked with connectives, is a challenging task. Traditional approaches for implicit DRR in Chinese have focused on exploring the concepts and features of words; however, these approaches have only yielded slow progress. Moreover, the lack of Chinese labeled data makes it more difficult to complete this task with high accuracy. To address this issue, we propose a novel hybrid DRR model combining a pretrained language model, namely bidirectional encoder representations from transformers (BERT), with recurrent neural networks. We use BERT as a text representation and pretraining model. In addition, we apply a tree structure to the implicit DRR in Chinese to produce hierarchical classes. The 19-class F1 score of our proposed method can reach 74.47% on the HIT-CIR Chinese discourse relation corpus. The attained results showed that the use of BERT and the proposed tree structure forms a novel and precise method that can automatically recognize the implicit relations of Chinese discourse.

[1]  Daniel Kondratyuk,et al.  Cross-Lingual Lemmatization and Morphology Tagging with Two-Stage Multilingual BERT Fine-Tuning , 2019, Proceedings of the 16th Workshop on Computational Research in Phonetics, Phonology, and Morphology.

[2]  Yang Liu,et al.  Recognizing Implicit Discourse Relations via Repeated Reading: Neural Networks with Multi-Level Attention , 2016, EMNLP.

[3]  Fang Kong,et al.  Building Chinese Discourse Corpus with Connective-driven Dependency Tree Structure , 2014, EMNLP.

[4]  Fang Kong,et al.  A CDT-Styled End-to-End Chinese Discourse Parser , 2016, NLPCC/ICCPOL.

[5]  Rashmi Prasad,et al.  Towards an Annotated Corpus of Discourse Relations in Hindi , 2008, IJCNLP.

[6]  Bohdan Didenko,et al.  Multi-headed Architecture Based on BERT for Grammatical Errors Correction , 2019, BEA@ACL.

[7]  Nianwen Xue,et al.  Robust Non-Explicit Neural Discourse Parser in English and Chinese , 2016, CoNLL Shared Task.

[8]  Katja Markert,et al.  The Leeds Arabic Discourse Treebank: Annotating Discourse Connectives for Arabic , 2010, LREC.

[9]  Jacob Eisenstein,et al.  Discourse Connectors for Latent Subjectivity in Sentiment Analysis , 2013, NAACL.

[10]  Jason Baldridge,et al.  Discourse Connective Argument Identification with Connective Specific Rankers , 2008, 2008 IEEE International Conference on Semantic Computing.

[11]  Yu Zhou,et al.  A Novel Translation Framework Based on Rhetorical Structure Theory , 2013, ACL.

[12]  Lung-Hao Lee,et al.  NCUEE at MEDIQA 2019: Medical Text Inference Using Ensemble BERT-BiLSTM-Attention Model , 2019, BioNLP@ACL.

[13]  Junji Tomita,et al.  A Simple but Effective Method to Incorporate Multi-turn Context with BERT for Conversational Machine Comprehension , 2019, Proceedings of the First Workshop on NLP for Conversational AI.

[14]  Ross B. Girshick,et al.  Focal Loss for Dense Object Detection , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Jiajun Zhang,et al.  Implicit Discourse Relation Recognition for English and Chinese with Multiview Modeling and Effective Representation Learning , 2017, ACM Trans. Asian Low Resour. Lang. Inf. Process..

[16]  Muhammad Abdul-Mageed,et al.  No Army, No Navy: BERT Semi-Supervised Learning of Arabic Dialects , 2019, WANLP@ACL 2019.

[17]  Fang Kong,et al.  Topic Tensor Network for Implicit Discourse Relation Recognition in Chinese , 2019, ACL.

[18]  Yuan Luo,et al.  Traditional Chinese medicine clinical records classification with BERT and domain specific corpora , 2019, J. Am. Medical Informatics Assoc..

[19]  Daniel Marcu,et al.  Building a Discourse-Tagged Corpus in the Framework of Rhetorical Structure Theory , 2001, SIGDIAL Workshop.

[20]  Zhang Muy,et al.  Chinese Discourse Relation Semantic Taxonomy and Annotation , 2014 .

[21]  Bonnie L. Webber,et al.  A Discourse Resource for Turkish: Annotating Discourse Connectives in the METU Corpus , 2008, IJCNLP.

[22]  Yuping Zhou,et al.  PDTB-style Discourse Annotation of Chinese Text , 2012, ACL.

[23]  Joyce Yue Chai,et al.  Discourse processing for context question answering based on linguistic knowledge , 2007, Knowl. Based Syst..

[24]  Yunfang Wu,et al.  Chinese Discourse Relation Recognition Using Parallel Corpus , 2013, 2013 Ninth International Conference on Computational Intelligence and Security.

[25]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[26]  Christian Chiarcos,et al.  A Recurrent Neural Model with Attention for the Recognition of Chinese Implicit Discourse Relations , 2017, ACL.

[27]  Long Tian,et al.  Combining Convolution Neural Network and Bidirectional Gated Recurrent Unit for Sentence Semantic Classification , 2018, IEEE Access.

[28]  Daniel Marcu,et al.  An Unsupervised Approach to Recognizing Discourse Relations , 2002, ACL.

[29]  Ao Feng,et al.  Target-Dependent Sentiment Classification With BERT , 2019, IEEE Access.

[30]  Ani Nenkova,et al.  Automatic sense prediction for implicit discourse relations in text , 2009, ACL.

[31]  Livio Robaldo,et al.  The Penn Discourse TreeBank 2.0. , 2008, LREC.

[32]  Nianwen Xue,et al.  A Systematic Study of Neural Discourse Models for Implicit Discourse Relation , 2017, EACL.