Figurative Language in Recognizing Textual Entailment

We introduce a collection of recognizing textual entailment (RTE) datasets focused on figurative language. We leverage five existing datasets annotated for a variety of figurative language – simile, metaphor, and irony – and frame them into over 12,500 RTE examples.We evaluate how well state-of-the-art models trained on popular RTE datasets capture different aspects of figurative language. Our results and analyses indicate that these models might not sufficiently capture figurative language, struggling to perform pragmatic inference and reasoning about world knowledge. Ultimately, our datasets provide a challenging testbed for evaluating RTE models.

[1]  Ekaterina Shutova,et al.  Computational approaches to figurative language , 2011 .

[2]  Yoav Goldberg,et al.  Breaking NLI Systems with Sentences that Require Simple Lexical Inferences , 2018, ACL.

[3]  O. Gelo,et al.  Unconventional metaphors and emotional-cognitive regulation in a metacognitive interpersonal therapy , 2012, Psychotherapy research : journal of the Society for Psychotherapy Research.

[4]  Debanjan Ghosh.  An empirical study of verbal irony , 2018 .

[5]  Robert A. Harris,et al.  Figurative language , 2021, The Craft of Poetry.

[6]  Adina Williams,et al.  Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition , 2020, ACL.

[7]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[8]  Gerard J. Steen,et al.  A method for linguistic metaphor identification : from MIP to MIPVU , 2010 .

[9]  S. Glucksberg.  Understanding Metaphors , 1998 .

[10]  Chris Callison-Burch,et al.  Most "babies" are "little" and most "problems" are "huge": Compositional Entailment in Adjective-Nouns , 2016, ACL.

[11]  Holger Schwenk,et al.  Supervised Learning of Universal Sentence Representations from Natural Language Inference Data , 2017, EMNLP.

[12]  W. Kintsch,et al.  Metaphor Comprehension: What Makes a Metaphor Difficult to Understand? , 2002 .

[13]  Kentaro Inui,et al.  Do Neural Models Learn Systematicity of Monotonicity Inference in Natural Language? , 2020, ACL.

[14]  John A. Barnden,et al.  Textual Entailment as an Evaluation Framework for Metaphor Resolution: A Proposal , 2008, STEP.

[15]  Roi Reichart,et al.  Sarcasm SIGN: Interpreting Sarcasm with Sentiment Based Monolingual Machine Translation , 2017, ACL.

[16]  Smaranda Muresan,et al.  Generating similes effortlessly like a Pro: A Style Transfer Approach for Simile Generation , 2020, EMNLP.

[17]  Rachel Rudinger,et al.  Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation , 2018, BlackboxNLP@EMNLP.

[18]  Raymond W. Gibbs,et al.  Beyond the Lexicon: Creativity in Language Production , 1988 .

[19]  Carlo Strapparava,et al.  Metaphor: A Computational Perspective by Tony Veale, Ekaterina Shutova and Beata Beigman Klebanov , 2016, CL.

[20]  Alex Wang,et al.  Probing What Different NLP Tasks Teach Machines about Function Word Comprehension , 2019, *SEMEVAL.

[21]  Kevin Duh,et al.  Inference is Everything: Recasting Semantic Resources into a Unified Evaluation Framework , 2017, IJCNLP.

[22]  Christopher Potts,et al.  A large annotated corpus for learning natural language inference , 2015, EMNLP.

[23]  Jean-Philippe Bernardy,et al.  What Kind of Natural Language Inference are NLP Systems Learning: Is this Enough? , 2019, ICAART.

[24]  Smaranda Muresan,et al.  MERMAID: Metaphor Generation with Symbolism and Discriminative Decoding , 2021, NAACL.

[25]  Noah D. Goodman,et al.  Evaluating Compositionality in Sentence Embeddings , 2018, CogSci.

[26]  Saif Mohammad,et al.  Metaphor as a Medium for Emotion: An Empirical Study , 2016, *SEMEVAL.

[27]  Stephen Pulman,et al.  Using the Framework , 1996 .

[28]  R. Kreuz,et al.  Why Do People Use Figurative Language? , 1994 .

[29]  Benjamin Van Durme,et al.  Temporal Reasoning in Natural Language Inference , 2020, FINDINGS.

[30]  Debanjan Ghosh,et al.  "With 1 follower I must be AWESOME : P". Exploring the role of irony markers in irony recognition , 2018, ICWSM.

[31]  D. Sperber,et al.  Irony and the Use-Mention Distinction , 1981 .

[32]  Benjamin Van Durme,et al.  Probing Neural Language Models for Human Tacit Assumptions , 2020, CogSci.

[33]  Roy Bar-Haim,et al.  The Second PASCAL Recognising Textual Entailment Challenge , 2006 .

[34]  Lysandre Debut,et al.  HuggingFace's Transformers: State-of-the-art Natural Language Processing , 2019, ArXiv.

[35]  Mark Steedman,et al.  Temporal and Aspectual Entailment , 2019, IWCS.

[36]  Debanjan Ghosh,et al.  Interpreting Verbal Irony: Linguistic Strategies and the Connection to the Type of Semantic Incongruity , 2020, SCIL.

[37]  Julia Jorgensen,et al.  The functions of sarcastic irony in speech , 1996 .

[38]  Ido Dagan,et al.  The Third PASCAL Recognizing Textual Entailment Challenge , 2007, ACL-PASCAL@ACL.

[39]  Tyler Marghetis,et al.  Literal and Metaphorical Senses in Compositional Distributional Semantic Models , 2016, ACL.

[40]  Samuel R. Bowman,et al.  A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference , 2017, NAACL.

[41]  Véronique Hoste,et al.  SemEval-2018 Task 3: Irony Detection in English Tweets , 2018, *SEMEVAL.

[42]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[43]  Rodrigo Agerri,et al.  Metaphor in Textual Entailment , 2008, COLING.

[44]  Ellie Pavlick,et al.  How well do NLI models capture verb veridicality? , 2019, EMNLP.

[45]  Adam Poliak,et al.  A survey on Recognizing Textual Entailment as an NLP Evaluation , 2020, EVAL4NLP.

[46]  Susan R. Fussell,et al.  Figurative Language in Emotional Communication , 1998 .

[47]  Carolyn Penstein Rosé,et al.  Stress Test Evaluation for Natural Language Inference , 2018, COLING.

[48]  James H. Martin.  A corpus-based analysis of context effects on metaphor comprehension , 2005 .

[49]  E. Winner,et al.  Why not say it directly? The social functions of irony , 1995 .

[50]  Lawrence S. Moss,et al.  Probing Natural Language Inference Models through Semantic Fragments , 2020, AAAI.