Multi-word expressions in textual inference: Much ado about nothing?

Multi-word expressions (MWE) have seen much attention from the NLP community. In this paper, we investigate their impact on the recognition of textual entailment (RTE). Using the manual Microsoft Research annotations, we first manually count and classify MWEs in RTE data. We find few, most of which are arguably unlikely to cause processing problems. We then consider the impact of MWEs on a current RTE system. We are unable to confirm that entailment recognition suffers from wrongly aligned MWEs. In addition, MWE alignment is difficult to improve, since MWEs are poorly represented in state-of-the-art paraphrase resources, the only available sources for multi-word similarities. We conclude that RTE should concentrate on other phenomena impacting entailment, and that paraphrase knowledge is best understood as capturing general lexico-syntactic variation.

[1]  Guillaume Cleuziou,et al.  Unsupervised Learning of General-Specific Noun Relations from the Web , 2008, FLAIRS Conference.

[2]  Christopher D. Manning,et al.  Natural Logic for Textual Inference , 2007, ACL-PASCAL@ACL.

[3]  Christiane Fellbaum,et al.  Book Reviews: WordNet: An Electronic Lexical Database , 1999, CL.

[4]  Mark Dras,et al.  Entailment due to Syntactically Encoded Semantic Relationships , 2007, ALTA.

[5]  Marc Sebban,et al.  Learning probabilistic models of tree edit distance , 2008, Pattern Recognit..

[6]  Lucy Vanderwende,et al.  What Syntax Can Contribute in the Entailment Task , 2005, MLCW.

[7]  Chris Callison-Burch,et al.  Paraphrasing with Bilingual Parallel Corpora , 2005, ACL.

[8]  Hua Wu,et al.  Pivot language approach for phrase-based statistical machine translation , 2007, ACL.

[9]  Aurélien Max.  Local Rephrasing Suggestions for Supporing the Work of Writers , 2008, GoTAL.

[10]  Regina Barzilay,et al.  Learning to Paraphrase: An Unsupervised Approach Using Multiple-Sequence Alignment , 2003, NAACL.

[11]  Christopher D. Manning,et al.  A Phrase-Based Alignment Model for Natural Language Inference , 2008, EMNLP.

[12]  Carolyn Penstein Rosé,et al.  A Hybrid Text Classification Approach for Analysis of Student Essays , 2003, HLT-NAACL 2003.

[13]  Katrin Erk,et al.  Paraphrase Assessment in Structured Vector Space: Exploring Parameters and Datasets , 2009 .

[14]  Marcello Federico,et al.  Phrase-based statistical machine translation with pivot languages. , 2008, IWSLT.

[15]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[16]  Ido Dagan,et al.  PROBABILISTIC TEXTUAL ENTAILMENT: GENERIC APPLIED MODELING OF LANGUAGE VARIABILITY , 2004 .

[17]  Mauro Birattari,et al.  Swarm Intelligence , 2012, Lecture Notes in Computer Science.

[18]  Hermann Ney,et al.  Statistical multi-source translation , 2001, MTSUMMIT.

[19]  Roy Bar-Haim,et al.  Definition and Analysis of Intermediate Entailment Levels , 2005, EMSEE@ACL.

[20]  Stanley M. Selkow,et al.  The Tree-to-Tree Editing Problem , 1977, Inf. Process. Lett..

[21]  Daniel Jurafsky,et al.  Learning Syntactic Patterns for Automatic Hypernym Discovery , 2004, NIPS.

[22]  Philippe Muller,et al.  A Qualitative Theory of Motion Based on Spatio-Temporal Primitives , 1998, KR.

[23]  Peter W. Foltz,et al.  Automated Essay Scoring: Applications to Educational Technology , 1999 .

[24]  Vasile Rus,et al.  Logic Form Transformation of WordNet and its Applicability to Question Answering , 2001, ACL.

[25]  Elena Cabrio,et al.  Combining Specialized Entailment Engines for RTE-4 , 2008, TAC.

[26]  C. Condoravdi,et al.  Computing relative polarity for textual inference , 2006 .

[27]  Ming Zhou,et al.  Combining Multiple Resources to Improve SMT-based Paraphrasing Model , 2008, ACL.

[28]  Christopher D. Manning,et al.  Aligning Semantic Graphs for Textual Inference and Machine Reading , 2007 .

[29]  Chris Quirk,et al.  Monolingual Machine Translation for Paraphrase Generation , 2004, EMNLP.

[30]  B. Magnini,et al.  Recognizing Textual Entailment with Tree Edit Distance Algorithms , 2005 .

[31]  Satoshi Sekine,et al.  Automatic Paraphrase Discovery based on Context and Keywords between NE Pairs , 2005, IJCNLP.

[32]  B. Magnini,et al.  Tree Edit Distance for Recognizing Textual Entailment : Estimating the Cost of Insertion , 2006 .

[33]  John B. Lowe,et al.  The Berkeley FrameNet Project , 1998, ACL.

[34]  Patrick Pantel,et al.  Automatically Labeling Semantic Classes , 2004, NAACL.

[35]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques with Java implementations , 2002, SGMD.

[36]  Brian Young,et al.  The Cross-Breeding of Dictionaries , 2004, LREC.

[37]  Katrin Erk,et al.  A Structured Vector Space Model for Word Meaning in Context , 2008, EMNLP.

[38]  Philip N. Klein,et al.  Computing the Edit-Distance between Unrooted Ordered Trees , 1998, ESA.

[39]  Dekang Lin,et al.  Principle-Based Parsing Without Overgeneration , 1993, ACL.

[40]  Takenobu Tokunaga,et al.  Proceedings of the ACL 2003 workshop on Multiword expressions: analysis, acquisition and treatment - Volume 18 , 2003 .

[41]  Timothy Baldwin,et al.  Multiword Expressions: A Pain in the Neck for NLP , 2002, CICLing.

[42]  Martha Palmer,et al.  Class-Based Construction of a Verb Lexicon , 2000, AAAI/IAAI.

[43]  Philippe Langlais,et al.  Explorations in using grammatical dependencies for contextual phrase translation disambiguation , 2008, EAMT.

[44]  Marti A. Hearst.  Automatic Acquisition of Hyponyms from Large Text Corpora , 1992, COLING.

[45]  Chris Quirk,et al.  Unsupervised Construction of Large Paraphrase Corpora: Exploiting Massively Parallel News Sources , 2004, COLING.

[46]  Mirella Lapata,et al.  Vector-based Models of Semantic Composition , 2008, ACL.

[47]  Patrick Pantel,et al.  Towards Terascale Semantic Acquisition , 2004, COLING.

[48]  Stephen Pulman,et al.  Automarking: using computational linguistics to score short‚ free−text responses , 2003 .

[49]  Dekang Lin,et al.  DIRT – Discovery of Inference Rules from Text , 2001 .

[50]  Andrew Hickl,et al.  Recognizing Textual Entailment with LCC’s G ROUNDHOG System , 2005 .

[51]  Jean-Pierre Chanod,et al.  Robustness beyond shallowness: incremental deep parsing , 2002, Natural Language Engineering.

[52]  Adam Kilgarriff,et al.  Distributional Thesaurus vs. WordNet: A Comparison of Backoff Techniques for Unsupervised PP Attachment1 , 2006 .

[53]  M. A. R T H A P A L,et al.  Making fine-grained and coarse-grained sense distinctions , both manually and automatically , 2005 .

[54]  George A. Miller,et al.  WordNet: A Lexical Database for English , 1995, HLT.

[55]  Tom Mitchell,et al.  Towards robust computerised marking of free-text responses , 2002 .

[56]  Dekang Lin,et al.  Automatic Retrieval and Clustering of Similar Words , 1998, ACL.

[57]  Chris Brockett,et al.  Aligning the RTE 2006 Corpus , 2007 .

[58]  Keinosuke Fukunaga,et al.  Introduction to statistical pattern recognition (2nd ed.) , 1990 .

[59]  Daniel Marcu,et al.  Syntax-based Alignment of Multiple Translations: Extracting Paraphrases and Generating New Sentences , 2003, NAACL.

[60]  Regina Barzilay,et al.  Using Lexical Chains for Text Summarization , 1997 .

[61]  John Blackmore,et al.  Proceedings of the Twenty-Second International FLAIRS Conference (2009) c-rater:Automatic Content Scoring for Short Constructed Responses , 2022 .

[62]  Michael Collins,et al.  Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms , 2002, EMNLP.

[63]  M. Pennacchiotti,et al.  Learning Selectional Preferences for Entailment or Paraphrasing Rules , 2007 .

[64]  D. Roth,et al.  Context Sensitive Paraphrasing with a Single Unsupervised Classifier , 2007 .

[65]  Patrick Pantel,et al.  ISP: Learning Inferential Selectional Preferences , 2007, NAACL.

[66]  Stephen Pulman,et al.  Auto−marking 2: An update on the UCLES−Oxford University research into using computational linguistics to score short‚ free text responses , 2004 .

[67]  C. J. van Rijsbergen,et al.  The geometry of information retrieval , 2004 .

[68]  Christopher D. Manning,et al.  Robust Graph Alignment Methods for Textual Inference and Machine Reading , 2007, AAAI Spring Symposium: Machine Reading.

[69]  Patrick Pantel,et al.  Discovery of inference rules for question-answering , 2001, Natural Language Engineering.

[70]  Christopher D. Manning,et al.  An extended model of natural logic , 2009, IWCS.

[71]  Bonnie Lynn Webber,et al.  Question Answering based on Semantic Roles , 2007, ACL 2007.

[72]  F. Wilcoxon.  Individual Comparisons by Ranking Methods , 1945 .

[73]  Roy Bar-Haim,et al.  The Second PASCAL Recognising Textual Entailment Challenge , 2006 .

[74]  Ido Dagan,et al.  Scaling Web-based Acquisition of Entailment Relations , 2004, EMNLP.

[75]  Alexander S. Yeh,et al.  More accurate tests for the statistical significance of result differences , 2000, COLING.

[76]  Christopher D. Manning,et al.  Modeling Semantic Containment and Exclusion in Natural Language Inference , 2008, COLING.

[77]  Dan I. Moldovan,et al.  Question Answering with Lexical Chains Propagating Verb Arguments , 2006, ACL.

[78]  Daniel Jurafsky,et al.  Semantic Taxonomy Induction from Heterogenous Evidence , 2006, ACL.

[79]  Georgiana Dinu,et al.  Inference Rules and their Application to Recognizing Textual Entailment , 2009, EACL.

[80]  Adrian Iftene,et al.  Hypothesis Transformation and Semantic Variability Rules Used in Recognizing Textual Entailment , 2007, ACL-PASCAL@ACL.

[81]  Jerry R. Hobbs,et al.  Spatial Representation and Reasoning , 2006 .

[82]  Katja Hofmann,et al.  Automatic Extraction of Dutch Hypernym-Hyponym Pairs , 2007, CLIN 2007.

[83]  Ido Dagan,et al.  Instance-based Evaluation of Entailment Rule Acquisition , 2007, ACL.

[84]  Elif Yamangil,et al.  Mining Wikipedia's Article Revision History for Training Computational Linguistics Algorithms , 2008 .

[85]  Arul Menezes,et al.  Syntactic Contributions in the Entailment Task: an implementation , 2005 .

[86]  Farid Melgani,et al.  Classification of Electrocardiogram Signals With Support Vector Machines and Particle Swarm Optimization , 2008, IEEE Transactions on Information Technology in Biomedicine.

[87]  Peter Clark,et al.  The Seventh PASCAL Recognizing Textual Entailment Challenge , 2011, TAC.

[88]  Alessandro Moschitti,et al.  Shallow Semantic in Fast Textual Entailment Rule Learners , 2007, ACL-PASCAL@ACL.

[89]  Christiane Fellbaum,et al.  Augmenting WordNet for Deep Understanding of Text , 2008, STEP.

[90]  Robert Stalnaker,et al.  Presuppositions of Compound Sentences , 2008 .

[91]  Ralph Grishman,et al.  NOMLEX: a lexicon of nominalizations , 1998 .

[92]  Ido Dagan,et al.  The Third PASCAL Recognizing Textual Entailment Challenge , 2007, ACL-PASCAL@ACL.

[93]  Stephen G. Pulman,et al.  Information Extraction and Machine Learning: Auto-Marking Short Free Text Responses to Science Questions , 2005, AIED.

[94]  Mirella Lapata,et al.  Detecting Novel Compounds: The Role of Distributional Evidence , 2003, EACL.

[95]  Stefan Thater,et al.  A Semantic Approach To Textual Entailment: System Evaluation and Task Analysis , 2007, ACL-PASCAL@ACL.

[96]  Mirella Lapata,et al.  ParaMetric: An Automatic Evaluation Metric for Paraphrasing , 2008, COLING.

[97]  Horst Bunke,et al.  A probabilistic approach to learning costs for graph edit distance , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[98]  Ralph Grishman,et al.  Using NOMLEX to Produce Nominalization Patterns for Information Extraction , 1998, ACL 1998.

[99]  Martin Chodorow,et al.  C-rater: Automated Scoring of Short-Answer Questions , 2003, Comput. Humanit..

[100]  Christopher D. Manning,et al.  Introduction to Information Retrieval , 2010, J. Assoc. Inf. Sci. Technol..

[101]  Haifeng Wang,et al.  Pivot Approach for Extracting Paraphrase Patterns from Bilingual Corpora , 2008, ACL.

[102]  Andy Way,et al.  Exploiting source similarity for SMT using context-informed features , 2007, TMI.

[103]  Chris Callison-Burch,et al.  Syntactic Constraints on Paraphrases Extracted from Parallel Corpora , 2008, EMNLP.

[104]  Ludovic Denoyer,et al.  The Wikipedia XML Corpus , 2006, INEX.

[105]  Kuo-Chung Tai,et al.  The Tree-to-Tree Correction Problem , 1979, JACM.

[106]  Constantino Carlos Reyes-Aldasoro,et al.  The Bhattacharyya space for feature selection and its application to texture segmentation , 2006, Pattern Recognit..

[107]  Roberto Navigli,et al.  SemEval-2007 Task 10: English Lexical Substitution Task , 2007, Fourth International Workshop on Semantic Evaluations (SemEval-2007).

[108]  Philipp Koehn,et al.  Europarl: A Parallel Corpus for Statistical Machine Translation , 2005, MTSUMMIT.

[109]  Sanda M. Harabagiu,et al.  The Informative Role of WordNet in Open-Domain Question Answering , 2004, HLT-NAACL 2004.

[110]  Svetlana Stoyanchev,et al.  Predicting Concept Types in User Corrections in Dialog , 2009 .

[111]  Kazuaki Kishida.  Property of average precision and its generalization: An examination of evaluation indicator for information retrieval experiments , 2005 .

[112]  Stefan Evert,et al.  Proceedings of the Workshop on Multiword Expressions: Identifying and Exploiting Underlying Properties , 2006 .

[113]  Dan I. Moldovan,et al.  Lexical Chains for Question Answering , 2002, COLING.

[114]  Rada Mihalcea,et al.  Using WordNet and Lexical Operators to Improve Internet Searches , 2000, IEEE Internet Comput..

[115]  Kaizhong Zhang,et al.  Simple Fast Algorithms for the Editing Distance Between Trees and Related Problems , 1989, SIAM J. Comput..

[116]  Emiel Krahmer,et al.  Dependency-based paraphrasing for recognizing textual entailment , 2007, ACL-PASCAL@ACL.

[117]  John Cocke,et al.  A Statistical Approach to Machine Translation , 1990, CL.

[118]  Mary Gardiner,et al.  Practical Queries of a Massive n-gram Database , 2007, ALTA.

[119]  Fabian M. Suchanek,et al.  Yago: A Core of Semantic Knowledge Unifying WordNet and Wikipedia , 2007 .

[120]  Dekang Lin,et al.  Dependency-Based Evaluation of Minipar , 2003 .

[121]  J. Lowe,et al.  A Frame-Semantic Approach to Semantic Annotation , 1997 .

[122]  Ido Dagan,et al.  The Distributional Inclusion Hypotheses and Lexical Entailment , 2005, ACL.

[123]  Christopher D. Manning,et al.  Learning to recognize features of valid textual entailments , 2006, NAACL.

[124]  Andrew Hickl,et al.  A Discourse Commitment-Based Framework for Recognizing Textual Entailment , 2007, ACL-PASCAL@ACL.

[125]  Robert L. Mercer,et al.  The Mathematics of Statistical Machine Translation: Parameter Estimation , 1993, CL.

[126]  Ido Dagan,et al.  Contextual Preferences , 2008, ACL.

[127]  Rada Mihalcea,et al.  Text-to-Text Semantic Similarity for Automatic Short Answer Grading , 2009, EACL.

[128]  Marco Pennacchiotti,et al.  FATE: a FrameNet-Annotated Corpus for Textual Entailment , 2008, LREC.

[129]  Dan Klein,et al.  Fast Exact Inference with a Factored Model for Natural Language Parsing , 2002, NIPS.

[130]  Erik D. Demaine,et al.  An optimal decomposition algorithm for tree edit distance , 2006, TALG.

[131]  Rahul Bhagat,et al.  Large Scale Acquisition of Paraphrases for Learning Surface Patterns , 2008, ACL.

[132]  Ido Dagan,et al.  Learning Entailment Rules for Unary Templates , 2008, COLING.

[133]  Yanjun Ma,et al.  Using Supertags as Source Language Context in SMT , 2009, EAMT.

[134]  James Pustejovsky,et al.  Integrating Motion Predicate Classes with Spatial and Temporal Annotations , 2008, COLING.

[135]  K. Markert,et al.  When logical inference helps determining textual entailment ( and when it doesn ’ t ) , .

[136]  Beth Levin,et al.  English Verb Classes and Alternations: A Preliminary Investigation , 1993 .

[137]  Howard G. Tucker.  Mathematical methods in sample surveys , 1998 .