The Third PASCAL Recognizing Textual Entailment Challenge

This paper presents the Third PASCAL Recognising Textual Entailment Challenge (RTE-3), providing an overview of the dataset creating methodology and the submitted systems. In creating this year's dataset, a number of longer texts were introduced to make the challenge more oriented to realistic scenarios. Additionally, a pool of resources was offered so that the participants could share common tools. A pilot task was also set up, aimed at differentiating unknown entailments from identified contradictions and providing justifications for overall system decisions. 26 participants submitted 44 runs, using different approaches and generally presenting new entailment models and achieving higher scores than in the previous challenges.

[1]  S. Siegel,et al.  Nonparametric Statistics for the Behavioral Sciences , 2022, The SAGE Encyclopedia of Research Design.

[2]  J. Fleiss.  Measuring nominal scale agreement among many raters. , 1971 .

[3]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[4]  Fahiem Bacchus,et al.  Representing and reasoning with probabilistic knowledge , 1988 .

[5]  Joseph Y. Halpern.  An Analysis of First-Order Logics of Probability , 1989, IJCAI.

[6]  Fahiem Bacchus,et al.  Representing and reasoning with probabilistic knowledge - a logical approach to probabilities , 1991 .

[7]  Gennaro Chierchia,et al.  Meaning and Grammar: An Introduction to Semantics , 1990 .

[8]  John B. Lowe,et al.  The Berkeley FrameNet Project , 1998, ACL.

[9]  Ellen M. Voorhees,et al.  Overview of the Seventh Text REtrieval Conference , 1998 .

[10]  Peter F. Smith,et al.  Vagueness: A Reader , 1999 .

[11]  Gennaro Chierchia,et al.  Meaning and grammar (2nd ed.): an introduction to semantics , 2000 .

[12]  M. de Rijke,et al.  Light-Weight Entailment Checking for Computational Semantics , 2001 .

[13]  Daniel G. Bobrow,et al.  Entailment, intensionality and text understanding , 2003, HLT-NAACL 2003.

[14]  Ido Dagan,et al.  Scaling Web-based Acquisition of Entailment Relations , 2004, EMNLP.

[15]  Ani Nenkova,et al.  Evaluating Content Selection in Summarization: The Pyramid Method , 2004, NAACL.

[16]  Ido Dagan,et al.  PROBABILISTIC TEXTUAL ENTAILMENT: GENERIC APPLIED MODELING OF LANGUAGE VARIABILITY , 2004 .

[17]  John D. Burger,et al.  Generating an Entailment Corpus from News Headlines , 2005, EMSEE@ACL.

[18]  Lucy Vanderwende,et al.  What Syntax Can Contribute in the Entailment Task , 2005, MLCW.

[19]  Ido Dagan,et al.  Web Based Probabilistic Textual Entailment , 2005 .

[20]  Roy Bar-Haim,et al.  Definition and Analysis of Intermediate Entailment Levels , 2005, EMSEE@ACL.

[21]  Rada Mihalcea,et al.  Measuring the Semantic Similarity of Texts , 2005, EMSEE@ACL.

[22]  Kathleen R. McKeown,et al.  Applying the Pyramid Method in DUC 2005 , 2005 .

[23]  Ido Dagan,et al.  A Lexical Alignment Model for Probabilistic Textual Entailment , 2005, MLCW.

[24]  Lauri Karttunen,et al.  Local Textual Inference: Can it be Defined or Circumscribed? , 2005, EMSEE@ACL.

[25]  Emiel Krahmer,et al.  Classification of Semantic Relations by Humans and Machines , 2005, EMSEE@ACL.

[26]  Roy Bar-Haim,et al.  The Second PASCAL Recognising Textual Entailment Challenge , 2006 .

[27]  Ido Dagan,et al.  Machine Learning Challenges, Evaluating Predictive Uncertainty, Visual Object Classification and Recognizing Textual Entailment, First PASCAL Machine Learning Challenges Workshop, MLCW 2005, Southampton, UK, April 11-13, 2005, Revised Selected Papers , 2006, MLCW.