NOPE: A Corpus of Naturally-Occurring Presuppositions in English

Understanding language requires grasping not only the overtly stated content, but also making inferences about things that were left unsaid. These inferences include presuppositions, a phenomenon by which a listener learns about new information through reasoning about what a speaker takes as given. Presuppositions require complex understanding of the lexical and syntactic properties that trigger them as well as the broader conversational context. In this work, we introduce the Naturally-Occurring Presuppositions in English (NOPE) Corpus to investigate the context-sensitivity of 10 different types of presupposition triggers and to evaluate machine learning models’ ability to predict human inferences. We find that most of the triggers we investigate exhibit moderate variability. We further find that transformer-based models draw correct inferences in simple cases involving presuppositions, but they fail to capture the minority of exceptional cases in which human judgments reveal complex interactions between context and triggers.

[1]  Andreas Vlachos,et al.  FEVER: a Large-scale Dataset for Fact Extraction and VERification , 2018, NAACL.

[2]  Marie-Catherine de Marneffe,et al.  Evaluating BERT for natural language inference: A case study on the CommitmentBank , 2019, EMNLP.

[3]  P. Schlenker.  Triggering Presuppositions , 2019, Glossa: a journal of general linguistics.

[4]  R. Thomas McCoy,et al.  Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference , 2019, ACL.

[5]  E. Chemla.  Similarity: towards a unified account of scalar implicatures, free choice permission and presupposition projection , 2008 .

[6]  Ellie Pavlick,et al.  Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering , 2021, ACL.

[7]  Dorit Abusch,et al.  Presupposition Triggering from Alternatives , 2010, J. Semant..

[8]  Mandy Simons,et al.  On the Conversational Basis of Some Presuppositions , 2013 .

[9]  Robert Stalnaker,et al.  Presuppositions of Compound Sentences , 2008 .

[10]  Judith Degen,et al.  Prior Beliefs Modulate Projection , 2021, Open Mind.

[11]  Jeroen Groenendijk,et al.  Dynamic predicate logic , 1991 .

[12]  J. Delin.  Presupposition and Shared Knowledge in It-Clefts. , 1995 .

[13]  A. Tversky,et al.  The framing of decisions and the psychology of choice. , 1981, Science.

[14]  Samuel R. Bowman,et al.  A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference , 2017, NAACL.

[15]  Gianluca Giorgolo,et al.  Conventional implicature , 2020, Enriched Meanings.

[16]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[17]  Beth Levin,et al.  English Verb Classes and Alternations: A Preliminary Investigation , 1993 .

[18]  Irene Heim,et al.  Presupposition Projection and the Semantics of Attitude Verbs , 1992, J. Semant..

[19]  Adina Williams,et al.  Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition , 2020, ACL.

[20]  Christopher Potts.  Presupposition and Implicature , 2015 .

[21]  H. Savin,et al.  The projection problem for presuppositions , 1971 .

[22]  Rob A. van der Sandt,et al.  Presupposition Projection as Anaphora Resolution , 1992, J. Semant..

[23]  Dorit Abusch,et al.  Lexical Alternatives as a Source of Pragmatic Presuppositions , 2002 .

[24]  Márta Abrusán,et al.  Predicting the presuppositions of soft triggers , 2011 .

[25]  Noah A. Smith,et al.  Evaluating Models’ Local Decision Boundaries via Contrast Sets , 2020, FINDINGS.

[26]  Judith Degen,et al.  How Projective is Projective Content? Gradience in Projectivity and At-issueness , 2018, J. Semant..

[27]  David I. Beaver,et al.  What projects and why , 2010 .

[28]  Benjamin Van Durme,et al.  Uncertain Natural Language Inference , 2020, ACL.

[29]  David I. Beaver.  Presupposition and Assertion in Dynamic Semantics , 2001 .

[30]  Lysandre Debut,et al.  HuggingFace's Transformers: State-of-the-art Natural Language Processing , 2019, ArXiv.

[31]  Napoleon Katsos,et al.  Experimental investigations of the typology of presupposition triggers , 2012 .

[32]  SHALOM LAPPIN,et al.  Presuppositional effects of strong determiners: a processing account , 1988 .

[33]  Andrew Kehler,et al.  Predicting Reference: What do Language Models Learn about Discourse Models? , 2020, EMNLP.

[34]  Yuxing Chen,et al.  Harnessing the linguistic signal to predict scalar inferences , 2019, ACL.

[35]  Gerald Gazdar,et al.  A solution to the projection problem , 1979 .

[36]  Judith Tonhauser,et al.  The CommitmentBank: Investigating projection in naturally occurring discourse , 2019 .

[37]  Allyson Ettinger,et al.  What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models , 2019, TACL.

[38]  Robin Cooper,et al.  The syntax and semantics of when-questions , 1982 .

[39]  M. Lyons.  Presupposition , 2021, Encyclopedia of Autism Spectrum Disorders.

[40]  Christopher C. Cummins,et al.  Backgrounding and accommodation of presuppositions: an experimental approach , 2013 .

[41]  Ellie Pavlick,et al.  How well do NLI models capture verb veridicality? , 2019, EMNLP.

[42]  Kevin Duh,et al.  Inference is Everything: Recasting Semantic Resources into a Unified Evaluation Framework , 2017, IJCNLP.

[43]  Christopher Potts,et al.  A large annotated corpus for learning natural language inference , 2015, EMNLP.

[44]  David I. Beaver,et al.  A Uniform Analysis of 'Before' and 'After' , 2003 .

[45]  Ellen F. Prince,et al.  On the Syntactic Marking of Presupposed Open Propositions , 1986 .

[46]  Jacopo Romoli,et al.  The Presuppositions of Soft Triggers are Obligatory Scalar Implicatures , 2015, J. Semant..

[47]  LAURI KARTTUNEN,et al.  PRESUPPOSITION AND LINGUISTIC CONTEXT , 1974 .

[48]  Mohit Bansal,et al.  Adversarial NLI: A New Benchmark for Natural Language Understanding , 2020, ACL.

[49]  Holger Schwenk,et al.  Supervised Learning of Universal Sentence Representations from Natural Language Inference Data , 2017, EMNLP.

[50]  Jianfeng Gao,et al.  DeBERTa: Decoding-enhanced BERT with Disentangled Attention , 2020, ICLR.

[51]  Tomas Mikolov,et al.  Advances in Pre-Training Distributed Word Representations , 2017, LREC.

[52]  Taylor Mahler.  The social component of the projection behavior of clausal complement contents , 2020 .

[53]  木村 和夫.  Pragmatics , 1997, Language Teaching.

[54]  Judy L. Delin,et al.  Properties of It-Cleft Presupposition , 1992, J. Semant..

[55]  Omer Levy,et al.  GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding , 2018, BlackboxNLP@EMNLP.

[56]  Orvokki Tellervo Heinämäki,et al.  Semantics of English temporal connectives , 1974 .

[57]  Omer Levy,et al.  SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems , 2019, NeurIPS.

[58]  Ido Dagan,et al.  The Third PASCAL Recognizing Textual Entailment Challenge , 2007, ACL-PASCAL@ACL.

[59]  Rachel Rudinger,et al.  Lexicosyntactic Inference in Neural Models , 2018, EMNLP.