SemEval-2014 Task 10: Multilingual Semantic Textual Similarity

In Semantic Textual Similarity, systems rate the degree of semantic equivalence between two text snippets. This year, the participants were challenged with new data sets for English, as well as the introduction of Spanish, as a new language in which to assess semantic similarity. For the English subtask, we exposed the systems to a diversity of testing scenarios, by preparing additional OntoNotesWordNet sense mappings and news headlines, as well as introducing new genres, including image descriptions, DEFT discussion forums, DEFT newswire, and tweet-newswire headline mappings. For Spanish, since, to our knowledge, this is the first time that official evaluations are conducted, we used well-formed text, by featuring sentences extracted from encyclopedic content and newswire. The annotations for both tasks leveraged crowdsourcing. The Spanish subtask engaged 9 teams participating with 22 system runs, and the English subtask attracted 15 teams with 38 system runs.

[1]  William H. Press,et al.  Numerical Recipes: The Art of Scientific Computing , 1987 .

[2]  Martha Palmer,et al.  Verb Semantics and Lexical Selection , 1994, ACL.

[3]  Bob Rehder,et al.  How Well Can Passage Meaning be Derived without Using Word Order? A Comparison of Latent Semantic Analysis and Humans , 1997 .

[4]  Martin Chodorow,et al.  Combining local context and wordnet similarity for word sense identification , 1998 .

[5]  Christiane Fellbaum,et al.  Combining Local Context and Wordnet Similarity for Word Sense Identification , 1998 .

[6]  Dekang Lin,et al.  An Information-Theoretic Definition of Similarity , 1998, ICML.

[7]  Christiane Fellbaum,et al.  Book Reviews: WordNet: An Electronic Lexical Database , 1999, CL.

[8]  Mitchell P. Marcus,et al.  OntoNotes: The 90% Solution , 2006, NAACL.

[9]  Cyrus Rashtchian,et al.  Collecting Image Annotations Using Amazon’s Mechanical Turk , 2010, Mturk@HLT-NAACL.

[10]  Juan Martínez-Romo,et al.  Disentangling categorical relationships through a graph of co-occurrences. , 2011, Physical review. E, Statistical, nonlinear, and soft matter physics.

[11]  Eneko Agirre,et al.  SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity , 2012, *SEMEVAL.

[12]  Francesco Pozzi,et al.  Exponential smoothing weighted correlations , 2012 .

[13]  A. Kilgarriff.  *SEM 2012: The First Joint Conference on Lexical and Computational Semantics , 2012 .

[14]  Jan Snajder,et al.  TakeLab: Systems for Measuring Semantic Text Similarity , 2012, *SEMEVAL.

[15]  Philipp Cimiano,et al.  Orthonormal Explicit Topic Analysis for Cross-Lingual Document Matching , 2013, EMNLP.

[16]  Heng Ji,et al.  Linking Tweets to News: A Framework to Enrich Short Text Data in Social Media , 2013, ACL.

[17]  Eneko Agirre,et al.  *SEM 2013 shared task: Semantic Textual Similarity , 2013, *SEMEVAL.

[18]  Quoc V. Le,et al.  Grounded Compositional Semantics for Finding and Describing Images with Sentences , 2014, TACL.

[19]  Darnes Vilariño Ayala,et al.  BUAP: Evaluating Features for Multilingual and Cross-Level Semantic Textual Similarity , 2014, SemEval@COLING.

[20]  Steven Bethard,et al.  DLS@CU: Sentence Similarity from Word Alignment , 2014, *SEMEVAL.

[21]  Stefan Evert,et al.  SemantiKLUE: Robust Semantic Similarity at Multiple Levels Using Maximum Weight Matching , 2014, *SEMEVAL.

[22]  Tommaso Caselli,et al.  FBK-TR: Applying SVM with Multiple Linguistic Features for Cross-Level Semantic Similarity , 2014, *SEMEVAL.

[23]  Miguel Rios.  UoW: Multi-task Learning Gaussian Process for Semantic Textual Similarity , 2014, SemEval@COLING.

[24]  Alexander F. Gelbukh,et al.  UNAL-NLP: Combining Soft Cardinality Features for Semantic Textual Similarity, Relatedness and Entailment , 2014, *SEMEVAL.

[25]  Davide Buscaldi,et al.  LIPN: Introducing a new Geographical Context Similarity Measure and a Statistical Similarity Measure based on the Bhattacharyya coefficient , 2014, SemEval@COLING.

[26]  Björn Gambäck,et al.  NTNU: Measuring Semantic Similarity with Sublexical Feature Representations and Soft Cardinality , 2014, *SEMEVAL.

[27]  Rafael Muñoz,et al.  UMCC_DLSI_SemSim: Multilingual System for Measuring Semantic Textual Similarity , 2014, *SEMEVAL.

[28]  Timothy W. Finin,et al.  Meerkat Mafia: Multilingual and Cross-Level Semantic Textual Similarity Systems , 2014, SemEval@COLING.

[29]  Steven Bethard,et al.  Back to Basics for Monolingual Alignment: Exploiting Word Similarity and Contextual Evidence , 2014, TACL.

[30]  Andy Way,et al.  RTM-DCU: Referential Translation Machines for Semantic Similarity , 2014, SemEval@COLING.

[31]  Anubhav Gupta,et al.  TeamZ: Measuring Semantic Textual Similarity for Spanish Using an Overlap-Based Approach , 2014, *SEMEVAL.