Understanding by Understanding Not: Modeling Negation in Language Models

Negation is a core construction in natural language. Despite being very successful on many tasks, state-of-the-art pre-trained language models often handle negation incorrectly. To improve language models in this regard, we propose to augment the language modeling objective with an unlikelihood objective that is based on negated generic sentences from a raw text corpus. By training BERT with the resulting combined objective we reduce the mean top 1 error rate to 4% on the negated LAMA dataset. We also see some improvements on the negated NLI benchmarks.

[1]  Carolyn Penstein Rosé,et al.  Stress Test Evaluation for Natural Language Inference , 2018, COLING.

[2]  Yiyun Zhao,et al.  How does BERT’s attention change when you fine-tune? An analysis methodology and a case study in negation scope , 2020, ACL.

[3]  Guodong Zhou,et al.  Unsupervised Negation Focus Identification with Word-Topic Graph Model , 2015, EMNLP.

[4]  Jian Zhang,et al.  SQuAD: 100,000+ Questions for Machine Comprehension of Text , 2016, EMNLP.

[5]  Dipanjan Das,et al.  BERT Rediscovers the Classical NLP Pipeline , 2019, ACL.

[6]  Guodong Zhou,et al.  Negation Focus Identification with Contextual Discourse Information , 2014, ACL.

[7]  Samuel R. Bowman,et al.  A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference , 2017, NAACL.

[8]  Bonnie L. Webber,et al.  Detecting negation scope is easy, except when it isn’t , 2017, EACL.

[9]  Edouard Grave,et al.  Colorless Green Recurrent Networks Dream Hierarchically , 2018, NAACL.

[10]  R. Thomas McCoy,et al.  Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference , 2019, ACL.

[11]  Eduardo Blanco,et al.  An Analysis of Natural Language Inference Benchmarks through the Lens of Negation , 2020, EMNLP.

[12]  Hinrich Schutze,et al.  Negated LAMA: Birds cannot fly , 2019, ArXiv.

[13]  Ido Dagan,et al.  The Third PASCAL Recognizing Textual Entailment Challenge , 2007, ACL-PASCAL@ACL.

[14]  Kathleen E. Hamilton,et al.  Predicting the Focus of Negation: Model and Error Analysis , 2020, ACL.

[15]  Jason Weston,et al.  Neural Text Generation with Unlikelihood Training , 2019, ICLR.

[16]  R. Thomas McCoy,et al.  Syntactic Data Augmentation Increases Robustness to Inference Heuristics , 2020, ACL.

[17]  Bonnie L. Webber,et al.  Neural Networks For Negation Scope Detection , 2016, ACL.

[18]  Christopher D. Manning,et al.  Stanza: A Python Natural Language Processing Toolkit for Many Human Languages , 2020, ACL.

[19]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[20]  Jacob Andreas,et al.  Good-Enough Compositional Data Augmentation , 2019, ACL.

[21]  Christophe Gravier,et al.  T-REx: A Large Scale Alignment of Natural Language with Knowledge Base Triples , 2018, LREC.

[22]  Ilya Sutskever,et al.  Language Models are Unsupervised Multitask Learners , 2019 .

[23]  Roser Morante,et al.  A Metalearning Approach to Processing the Scope of Negation , 2009, CoNLL.

[24]  Kai Zou,et al.  EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks , 2019, EMNLP.

[25]  Dieuwke Hupkes,et al.  Do Language Models Understand Anything? On the Ability of LSTMs to Understand Negative Polarity Items , 2018, BlackboxNLP@EMNLP.

[26]  Wei Lu,et al.  Learning with Structured Representations for Negation Scope Extraction , 2018, ACL.

[27]  Tal Linzen,et al.  Targeted Syntactic Evaluation of Language Models , 2018, EMNLP.

[28]  Xiang Li,et al.  Commonsense Knowledge Base Completion , 2016, ACL.

[29]  Qiaoming Zhu,et al.  Negative Focus Detection via Contextual Attention Mechanism , 2019, EMNLP.

[30]  Kenton Lee,et al.  A BERT Baseline for the Natural Questions , 2019, ArXiv.

[31]  Jonathan Berant,et al.  oLMpics-On What Language Model Pre-training Captures , 2019, Transactions of the Association for Computational Linguistics.

[32]  Hiroya Takamura,et al.  An Analysis of the Utility of Explicit Negative Examples to Improve the Syntactic Abilities of Neural Language Models , 2020, ACL.

[33]  Samuel R. Bowman,et al.  Grammatical Analysis of Pretrained Sentence Encoders with Acceptability Judgments , 2019, ArXiv.