Semeval-2012 Task 8: Cross-lingual Textual Entailment for Content Synchronization

This paper presents the first round of the task on Cross-lingual Textual Entailment for Content Synchronization, organized within SemEval-2012. The task was designed to promote research on semantic inference over texts written in different languages, targeting at the same time a real application scenario. Participants were presented with datasets for different language pairs, where multi-directional entailment relations ("forward", "backward", "bidirectional", "no_entailment") had to be identified. We report on the training and test data used for evaluation, the process of their creation, the participating systems (10 teams, 92 runs), the approaches adopted and the results achieved.

[1]  Michael E. Lesk,et al.  Automatic sense disambiguation using machine readable dictionaries: how to tell a pine cone from an ice cream cone , 1986, SIGDOC '86.

[2]  Hermann Ney,et al.  Improved Statistical Alignment Models , 2000, ACL.

[3]  Hermann Ney,et al.  A Systematic Comparison of Various Statistical Alignment Models , 2003, CL.

[4]  Ido Dagan,et al.  PROBABILISTIC TEXTUAL ENTAILMENT: GENERIC APPLIED MODELING OF LANGUAGE VARIABILITY , 2004 .

[5]  Alon Lavie,et al.  METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments , 2005, IEEvaluation@ACL.

[6]  Elena Cabrio,et al.  Towards Extensible Textual Entailment Engines: The EDITS Package , 2009, AI*IA.

[7]  Marcello Federico,et al.  Towards Cross-Lingual Textual Entailment , 2010, NAACL.

[8]  Alpár Perini.  Detecting Textual Entailment with Conditions on Directional Text Relatedness Scores Revisited , 2010, TAC.

[9]  Matteo Negri,et al.  Creating a Bi-lingual Entailment Corpus through Translations with Mechanical Turk: $100 for a 10-day Rush , 2010, Mturk@HLT-NAACL.

[10]  Matteo Negri,et al.  An Open-Source Package for Recognizing Textual Entailment , 2010, ACL.

[11]  Marcello Federico,et al.  Using Bilingual Parallel Corpora for Cross-Lingual Textual Entailment , 2011, ACL.

[12]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[13]  Francis M. Tyers,et al.  Apertium: a free/open-source platform for rule-based machine translation , 2011, Machine Translation.

[14]  Matteo Negri,et al.  Divide and Conquer: Crowdsourcing the Creation of Cross-Lingual Textual Entailment Corpora , 2011, EMNLP.

[15]  Christof Monz,et al.  CoSyne: a framework for multilingual content synchronization of wikis , 2011, Int. Sym. Wikis.

[16]  Darnes Vilariño Ayala,et al.  BUAP: Lexical and Semantic Similarity for Cross-lingual Textual Entailment , 2012, SemEval@NAACL-HLT.

[17]  Mikel L. Forcada,et al.  UAlacant: Using Online Machine Translation for Cross-Lingual Textual Entailment , 2012, SemEval@NAACL-HLT.

[18]  Sivaji Bandyopadhyay,et al.  JU_CSE_NLP: Language Independent Cross-lingual Textual Entailment System , 2012, SemEval@NAACL-HLT.

[19]  Luca Dini,et al.  CELI: An Experiment with Cross Language Textual Entailment , 2012, SemEval@NAACL-HLT.

[20]  Katharina Wäschle,et al.  HDU: Cross-lingual Textual Entailment with SMT Features , 2012, *SEMEVAL.

[21]  Alexander F. Gelbukh,et al.  Soft Cardinality + ML: Learning Adaptive Similarity Functions for Cross-lingual Textual Entailment , 2012, SemEval@NAACL-HLT.

[22]  Alpár Perini.  DirRelCond3: Detecting Textual Entailment Across Languages With Conditions On Directional Text Relatedness Scores , 2012, SemEval@NAACL-HLT.

[23]  Marcello Federico,et al.  Match without a Referee: Evaluating MT Adequacy without Reference Translations , 2012, WMT@NAACL-HLT.

[24]  Marcello Federico,et al.  Detecting Semantic Equivalence and Information Disparity in Cross-lingual Documents , 2012, ACL.

[25]  Qun Liu,et al.  ICT: A Translation based Method for Cross-lingual Textual Entailment , 2012, SemEval@NAACL-HLT.

[26]  Matteo Negri,et al.  Chinese Whispers: Cooperative Paraphrase Acquisition , 2012, LREC.

[27]  José Guilherme Camargo de Souza,et al.  FBK: Cross-Lingual Textual Entailment Without Translation , 2012, SemEval@NAACL-HLT.

[28]  Marco Turchi,et al.  ALTN: Word Alignment Features for Cross-lingual Textual Entailment , 2013, SemEval@NAACL-HLT.

[29]  Darnes Vilariño Ayala,et al.  BUAP: N-gram based Feature Evaluation for the Cross-Lingual Textual Entailment Task , 2013, SemEval@NAACL-HLT.

[30]  Alexander F. Gelbukh,et al.  SOFTCARDINALITY: Learning to Identify Directional Cross-Lingual Entailment from Cardinalities and SMT , 2013, SemEval@NAACL-HLT.

[31]  Luca Dini,et al.  Celi: EDITS and Generic Text Pair Classification , 2013, *SEMEVAL.

[32]  Timothy Baldwin,et al.  Umelb: Cross-lingual Textual Entailment with Word Alignment and String Similarity Features , 2013, SemEval@NAACL-HLT.