Report on the 10th IWSLT evaluation campaign

The paper overviews the tenth evaluation campaign organized by the IWSLT workshop. The 2013 evaluation offered multiple tracks on lecture transcription and translation based on the TED Talks corpus. In particular, this year IWSLT included two automatic speech recognition tracks, on English and German, three speech translation tracks, from English to French, English to German, and German to English, and three text translation track, also from English to French, English to German, and German to English. In addition to the official tracks, speech and text translation optional tracks were offered involving 12 other languages: Arabic, Spanish, Portuguese (B), Italian, Chinese, Polish, Persian, Slovenian, Turkish, Dutch, Romanian, Russian. Overall, 18 teams participated in the evaluation for a total of 217 primary runs submitted. All runs were evaluated with objective metrics on a current test set and two progress test sets, in order to compare the progresses against systems of the previous years. In addition, submissions of one of the official machine translation tracks were also evaluated with human post-editing.

[1]  Russell V. Lenth,et al.  Computer Intensive Methods for Testing Hypotheses: An Introduction , 1990 .

[2]  Lynette Hirschman,et al.  Evaluating Message Understanding Systems: An Analysis of the Third Message Understanding Conference (MUC-3) , 1993, CL.

[3]  Noriko Kando,et al.  Overview of the IWSLT04 evaluation campaign , 2004, IWSLT.

[4]  Daniel Jurafsky,et al.  Automatic Tagging of Arabic Text: From Raw Text to Base Phrase Chunks , 2004, NAACL.

[5]  Philipp Koehn,et al.  Europarl: A Parallel Corpus for Statistical Machine Translation , 2005, MTSUMMIT.

[6]  Daniel Jurafsky,et al.  A Conditional Random Field Word Segmenter for Sighan Bakeoff 2005 , 2005, IJCNLP.

[7]  Chiori Hori,et al.  Overview of the IWSLT 2005 Evaluation Campaign , 2005, IWSLT.

[8]  Stefan Riezler,et al.  On Some Pitfalls in Automatic Evaluation and Significance Testing for MT , 2005, IEEvaluation@ACL.

[9]  Michael Paul,et al.  Overview of the IWSLT06 evaluation campaign , 2006, IWSLT.

[10]  Matthew G. Snover,et al.  A Study of Translation Edit Rate with Targeted Human Annotation , 2006, AMTA.

[11]  Timothy R. Anderson,et al.  The MIT-LL/AFRL IWSLT-2006 MT system , 2006, IWSLT.

[12]  Philipp Koehn,et al.  Moses: Open Source Toolkit for Statistical Machine Translation , 2007, ACL.

[13]  Cameron S. Fordyce,et al.  Overview of the IWSLT 2007 evaluation campaign , 2007, IWSLT.

[14]  Mauro Cettolo,et al.  IRSTLM: an open source toolkit for handling large scale language models , 2008, INTERSPEECH.

[15]  Michael Paul,et al.  Overview of the IWSLT 2008 evaluation campaign. , 2008, IWSLT.

[16]  Michael Paul,et al.  Overview of the IWSLT 2009 evaluation campaign , 2009, IWSLT.

[17]  Jörg Tiedemann,et al.  News from OPUS — A collection of multilingual parallel corpora with tools and interfaces , 2009 .

[18]  Andreas Eisele,et al.  MultiUN: A Multilingual Corpus from United Nation Documents , 2010, LREC.

[19]  Hermann Ney,et al.  The RWTH Aachen Machine Translation System for WMT 2010 , 2010, IWSLT.

[20]  Sebastian Stüker,et al.  Overview of the IWSLT 2010 evaluation campaign , 2010, IWSLT.

[21]  Sebastian Stüker,et al.  Overview of the IWSLT 2011 evaluation campaign , 2011, IWSLT.

[22]  Hideki Kashioka,et al.  The NICT ASR system for IWSLT2011 , 2011, IWSLT.

[23]  Mauro Cettolo,et al.  WIT3: Web Inventory of Transcribed and Translated Talks , 2012, EAMT.

[24]  Eiichiro Sumita,et al.  The NICT translation system for IWSLT 2012 , 2012, IWSLT.

[25]  Sebastian Stüker,et al.  Overview of the IWSLT 2012 evaluation campaign , 2012, IWSLT.

[26]  Marcello Federico.  Measuring User Productivity in Machine Translation Enhanced Computer Assisted Translation , 2012, AMTA.

[27]  Alex Waibel,et al.  EU-BRIDGE MT: text translation of talks in the EU-BRIDGE project , 2013, IWSLT.

[28]  The CASIA machine translation system for IWSLT 2013 , 2013, IWSLT.

[29]  Stefan Riezler,et al.  The Heidelberg University Machine Translation Systems for IWSLT2013 , 2013 .

[30]  Hermann Ney,et al.  The RWTH Aachen German and English LVCSR systems for IWSLT-2013 , 2013 .

[31]  Jong-Hyeok Lee,et al.  A discriminative reordering parser for IWSLT 2013 , 2013, IWSLT.

[32]  Marcello Federico,et al.  FBK’s machine translation systems for the IWSLT 2013 evaluation campaign , 2013, IWSLT.

[33]  Kevin Duh,et al.  NTT-NAIST SMT Systems for IWSLT 2013 , 2013 .

[34]  Preslav Nakov,et al.  QCRI at IWSLT 2013: experiments in Arabic-English and English-Arabic spoken language translation , 2013, IWSLT.

[35]  Philipp Koehn,et al.  Findings of the 2013 Workshop on Statistical Machine Translation , 2013, WMT@ACL.

[36]  Jeffrey Heer,et al.  The efficacy of human post-editing for language translation , 2013, CHI.

[37]  D. Giuliani,et al.  FBK @ IWSLT 2013 – ASR tracks , 2013, IWSLT.

[38]  Ilknur Durgar El-Kahlout,et al.  TÜBİTAK Turkish-English submissions for IWSLT 2013 , 2013, IWSLT.

[39]  Dekai Wu,et al.  Improving machine translation into Chinese by tuning against Chinese MEANT , 2013, IWSLT.

[40]  Nadir Durrani,et al.  Edinburgh SLT and MT System Description for the IWSLT 2014 Evaluation , 2013 .

[41]  Ngoc-Quan Pham,et al.  The speech recognition and machine translation system of IOIT for IWSLT 2013 , 2013, IWSLT.

[42]  Peter Bell,et al.  Description of the UEDIN System for German ASR , 2013 .

[43]  Matthias Sperber,et al.  The 2013 KIT IWSLT speech-to-text systems for German and English , 2013, IWSLT.

[44]  Qin Gao,et al.  MSR-FBK IWSLT 2013 SLT system description , 2013, IWSLT.

[45]  Fergus McInnes,et al.  The UEDIN English ASR System for the IWSLT 2013 Evaluation , 2013 .

[46]  Jan Niehues,et al.  The KIT translation systems for IWSLT 2015 , 2015, IWSLT.

[47]  Graham Neubig,et al.  The NAIST English Speech Recognition System for IWSLT 2015 , 2013 .