Implicit Discourse Relation Classification: We Need to Talk about Evaluation

Implicit relation classification on Penn Discourse TreeBank (PDTB) 2.0 is a common benchmark task for evaluating the understanding of discourse relations. However, the lack of consistency in preprocessing and evaluation poses challenges to fair comparison of results in the literature. In this work, we highlight these inconsistencies and propose an improved evaluation protocol. Paired with this protocol, we report strong baseline results from pretrained sentence encoders, which set the new state-of-the-art for PDTB 2.0. Furthermore, this work is the first to explore fine-grained relation classification on PDTB 3.0. We expect our work to serve as a point of comparison for future work, and also as an initiative to discuss models of larger context and possible data augmentations for downstream transferability.

[1]  Ruihong Huang,et al.  Improving Implicit Discourse Relation Classification by Modeling Inter-dependencies of Discourse Units in a Paragraph , 2018, NAACL.

[2]  Ani Nenkova,et al.  Easily Identifiable Discourse Relations , 2008, COLING.

[3]  Zheng-Yu Niu,et al.  Multi-task Attention-based Neural Networks for Implicit Discourse Relationship Representation and Identification , 2017, EMNLP.

[4]  Wei Shi,et al.  Learning to Explicitate Connectives with Seq2Seq Network for Implicit Discourse Relation Classification , 2018, IWCS.

[5]  Jacob Eisenstein,et al.  One Vector is Not Enough: Entity-Augmented Distributed Semantics for Discourse Relations , 2014, TACL.

[6]  Ani Nenkova,et al.  Using entity features to classify implicit discourse relations , 2010, SIGDIAL Conference.

[7]  Wei Shi,et al.  Do We Need Cross Validation for Discourse Relation Classification? , 2017, EACL.

[8]  Hai Zhao,et al.  Adversarial Connective-exploiting Networks for Implicit Discourse Relation Classification , 2017, ACL.

[9]  Hai Zhao,et al.  Deep Enhanced Representation for Implicit Discourse Relation Recognition , 2018, COLING.

[10]  Hwee Tou Ng,et al.  The CoNLL-2015 Shared Task on Shallow Discourse Parsing , 2015, CoNLL.

[11]  Rachel Rudinger,et al.  Hypothesis Only Baselines in Natural Language Inference , 2018, *SEMEVAL.

[12]  Jian Su,et al.  Kernel Based Discourse Relation Recognition with Temporal Ordering Information , 2010, ACL.

[13]  Claire Cardie,et al.  Improving Implicit Discourse Relation Recognition Through Feature Set Optimization , 2012, SIGDIAL Conference.

[14]  Peter Jansen,et al.  Discourse Complements Lexical Semantics for Non-factoid Answer Reranking , 2014, ACL.

[15]  Kyle Gorman,et al.  We Need to Talk about Standard Splits , 2019, ACL.

[16]  Thien Huu Nguyen,et al.  Employing the Correspondence of Relations and Connectives to Identify Implicit Discourse Relations via Label Embeddings , 2019, ACL.

[17]  Rashmi Prasad,et al.  The Penn Discourse Treebank , 2004, LREC.

[18]  Nianwen Xue,et al.  A Systematic Study of Neural Discourse Models for Implicit Discourse Relation , 2017, EACL.

[19]  Yiming Yang,et al.  XLNet: Generalized Autoregressive Pretraining for Language Understanding , 2019, NeurIPS.

[20]  Pascal Denis,et al.  Comparing Word Representations for Implicit Discourse Relation Classification , 2015, EMNLP.

[21]  Junhan Zhao,et al.  Memorizing All for Implicit Discourse Relation Recognition , 2019, ArXiv.

[22]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[23]  Noah D. Goodman,et al.  DisSent: Learning Sentence Representations from Explicit Discourse Relations , 2019, ACL.

[24]  Yang Liu,et al.  Recognizing Implicit Discourse Relations via Repeated Reading: Neural Networks with Multi-Level Attention , 2016, EMNLP.

[25]  Hwee Tou Ng,et al.  Recognizing Implicit Discourse Relations in the Penn Discourse Treebank , 2009, EMNLP.

[26]  Xuanjing Huang,et al.  Implicit Discourse Relation Detection via a Deep Architecture with Gated Relevance Network , 2016, ACL.

[27]  Ani Nenkova,et al.  Automatic sense prediction for implicit discourse relations in text , 2009, ACL.

[28]  Min-Yen Kan,et al.  Linguistic Properties Matter for Implicit Discourse Relation Recognition: Combining Semantic Interaction, Topic Continuity and Attribution , 2018, AAAI.

[29]  조성현,et al.  Level of Detail , 2017, Encyclopedia of GIS.

[30]  Kshitij P. Fadnis,et al.  Doc2Dial: A Framework for Dialogue Composition Grounded in Documents , 2020, AAAI.

[31]  Andrew Kehler,et al.  Predicting the Presence of Discourse Connectives , 2013, EMNLP.

[32]  Guillaume Bouchard,et al.  Interpretation of Natural Language Rules in Conversational Machine Reading , 2018, EMNLP.

[33]  Junyi Jessy Li,et al.  Reducing Sparsity Improves the Recognition of Implicit Discourse Relations , 2014, SIGDIAL Conference.