Lost in translation: the problems of using mainstream MTevaluation metrics for sign language translation

In this paper we consider the problems of applying corpus-based techniques to minority languages that are neither politically recognised nor have a formally accepted writing system, namely sign languages. We discuss the adoption of an annotated form of sign language data as a suitable corpus for the development of a data-driven machine translation (MT) system, and deal with issues that arise from its use. Useful software tools that facilitate easy annotation of video data are also discussed. Furthermore, we address the problems of using traditional MT evaluation metrics for sign language translation. Based on the candidate translations produced from our example-based machine translation system, we discuss why standard metrics fall short of providing an accurate evaluation and suggest more suitable evaluation methods.

[1]  Andy Way,et al.  Robust large-scale EBMT with marker-based segmentation , 2004, TMI.

[2]  Dean Barker,et al.  South African Sign Language Machine Translation System , 2003, AFRIGRAPH '03.

[3]  Andy Way,et al.  An Example-Based Approach to Translating Sign Language , 2005, MTSUMMIT.

[4]  Andy Way,et al.  Treebank-Based Acquisition of Multilingual Unification Grammar Resources , 2005 .

[5]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[6]  Ian W. Marshall,et al.  Sign language generation using HPSG. , 2002, TMI.

[7]  Thomas R. G. Green,et al.  The necessity of syntax markers: Two experiments with artificial languages , 1979 .

[8]  Chin-Yew Lin,et al.  ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation , 2004, COLING.

[9]  Hermann Ney,et al.  Morpho-Syntax Based Statistical Methods for Automatic Sign Language Translation , 2006, EAMT.

[10]  Tony Veale,et al.  The Challenges of Cross-Modal Translation: English-to-Sign-Language Translation in the Zardoz System , 1998, Machine Translation.

[11]  Hermann Ney,et al.  Statistical Sign Language Translation , 2004 .

[12]  Andy Way,et al.  Long-Distance Dependency Resolution in Automatically Acquired Wide-Coverage PCFG-Based LFG Approximations , 2004, ACL.

[13]  Andy Way,et al.  Large-Scale Induction and Evaluation of Lexical Resources from the Penn-II and Penn-III Treebanks , 2005, Computational Linguistics.

[14]  Great Britain. Hm Factory Inspectorate.  An introductory guide , 1987 .

[15]  Philipp Koehn,et al.  Europarl: A Parallel Corpus for Statistical Machine Translation , 2005, MTSUMMIT.

[16]  Andy Way,et al.  Example-based controlled translation , 2004, EAMT.

[17]  Norman I. Badler,et al.  A machine translation system from English to American Sign Language , 2000, AMTA.

[18]  Matt Huenerfauth.  American Sign Language Generation: Multimodal NLG with Multiple Linguistic Channels , 2005, ACL.

[19]  SystemBritta,et al.  Towards an Automatic Sign Language Translation , 1999 .

[20]  P. Smith,et al.  Two Experiments with Artificial Languages , 1970 .

[21]  Alex Waibel,et al.  The CMU statistical machine translation system , 2003, MTSUMMIT.

[22]  Éva Sáfár,et al.  The Architecture of an English-Text-to-Sign-Languages Translation System , 2001 .

[23]  W. Stokoe.  Sign language structure: an outline of the visual communication systems of the American deaf. 1960. , 1961, Journal of deaf studies and deaf education.