Data-Driven Machine Translation for Sign Languages

This thesis explores the application of data-driven machine translation (MT) to sign languages (SLs). The provision of an SL MT system can facilitate communication between Deaf and hearing people by translating information into the native and preferred language of the individual. We begin with an introduction to SLs, focussing on Irish Sign Language - the native language of the Deaf in Ireland. We describe their linguistics and mechanics including similarities and differences with spoken languages. Given the lack of a formalised written form of these languages, an outline of annotation formats is discussed as well as the issue of data collection. We summarise previous approaches to SL MT, highlighting the pros and cons of each approach. Initial experiments in the novel area of example-based MT for SLs are discussed and an overview of the problems that arise when automatically translating these manual-visual languages is given. Following this we detail our data-driven approach, examining the MT system used and modifications made for the treatment of SLs and their annotation. Through sets of automatically evaluated experiments in both language directions, we consider the merits of data-driven MT for SLs and outline the mainstream evaluation metrics used. To complete the translation into SLs, we discuss the addition and manual evaluation of a signing avatar for real SL output.

[1]  Matt Huenerfauth.  American Sign Language Generation: Multimodal NLG with Multiple Linguistic Channels , 2005, ACL.

[2]  Andy Way,et al.  Robust large-scale EBMT with marker-based segmentation , 2004, TMI.

[3]  Scott K. Liddell,et al.  American Sign Language: The Phonological Base , 2013 .

[4]  C Neidle,et al.  SignStream: A tool for linguistic and computer vision research on visual-gestural language data , 2001, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[5]  H. Kamp.  A Theory of Truth and Semantic Representation , 2008 .

[6]  Harold L. Somers,et al.  An introduction to machine translation , 1992 .

[7]  SystemBritta,et al.  Towards an Automatic Sign Language Translation , 1999 .

[8]  B. Shneiderman.  Designing the User Interface (3rd Ed.) , 1998 .

[9]  James E. Powell.  Designing User Interfaces , 1991 .

[10]  Ralph Johnson,et al.  design patterns elements of reusable object oriented software , 2019 .

[11]  C. B. Traxler,et al.  The Stanford Achievement Test, 9th Edition: National Norming and Performance Standards for Deaf and Hard-of-Hearing Students. , 2000, Journal of deaf studies and deaf education.

[12]  Padraig Cunningham,et al.  Competitive Hypothesis Resolution in TWIG, a Blackboard-Driven Text Understanding System , 1992, ECAI.

[13]  STUART M. SHIEBER.  RESTRICTING THE WEAK‐GENERATIVE CAPACITY OF SYNCHRONOUS TREE‐ADJOINING GRAMMARS , 1994, Comput. Intell..

[14]  Franz Josef Och,et al.  Minimum Error Rate Training in Statistical Machine Translation , 2003, ACL.

[15]  Hermann Ney,et al.  Speech recognition techniques for a sign language recognition system , 2007, INTERSPEECH.

[16]  Hermann Ney,et al.  Improved Statistical Alignment Models , 2000, ACL.

[17]  Ian W. Marshall,et al.  Sign language generation using HPSG. , 2002, TMI.

[18]  Thomas R. G. Green,et al.  The necessity of syntax markers: Two experiments with artificial languages , 1979 .

[19]  Christiane Fellbaum,et al.  Book Reviews: WordNet: An Electronic Lexical Database , 1999, CL.

[20]  Mitchell P. Marcus,et al.  Generating american sign language classifier predicates for english-to-asl machine translation , 2006 .

[21]  George R. Doddington,et al.  The ATIS Spoken Language Systems Pilot Corpus , 1990, HLT.

[22]  Angus B. Grieve-Smith,et al.  SignSynth: A Sign Language Synthesis Application Using Web3D and Perl , 2001, Gesture Workshop.

[23]  Julie Carson-Berndsen,et al.  The Jess Blizzard Challenge 2007 Entry , 2006 .

[24]  Hermann Ney,et al.  Discriminative Training and Maximum Entropy Models for Statistical Machine Translation , 2002, ACL.

[25]  Norman I. Badler,et al.  A machine translation system from English to American Sign Language , 2000, AMTA.

[26]  Andy Way,et al.  MATREX: DCU machine translation system for IWSLT 2006. , 2006, IWSLT.

[27]  Andy Way,et al.  Hand in hand: automatic sign language to English translation , 2007, TMI.

[28]  L. Rabiner,et al.  An introduction to hidden Markov models , 1986, IEEE ASSP Magazine.

[29]  M. Baltin,et al.  The Mental representation of grammatical relations , 1985 .

[30]  Mark Aronoff,et al.  Contemporary linguistics: An introduction , 1989 .

[31]  Lorraine Leeson.  Aspects of verbal valency in Irish Sign Language , 2002 .

[32]  Charlotte Lee Baker-Shenk,et al.  American Sign Language : A Teacher's Resource Text on Grammar and Culture , 1991 .

[33]  Scott K. Liddell.  American Sign Language Syntax , 1981 .

[34]  Philipp Koehn,et al.  (Meta-) Evaluation of Machine Translation , 2007, WMT@ACL.

[35]  Van Nostrand,et al.  Error Bounds for Convolutional Codes and an Asymptotically Optimum Decoding Algorithm , 1967 .

[36]  Matt Huenerfauth,et al.  Spatial and planning models of ASL classifier predicates for machine translation , 2004, TMI.

[37]  Tony Veale,et al.  The Challenges of Cross-Modal Translation: English-to-Sign-Language Translation in the Zardoz System , 1998, Machine Translation.

[38]  Onno Crasborn,et al.  Sign language transcription conventions for the ECHO Project , 2003 .

[39]  Chung-Hsien Wu,et al.  Transfer-based statistical translation of Taiwanese sign language using PCFG , 2007, TALIP.

[40]  Philipp Koehn,et al.  Moses: Open Source Toolkit for Statistical Machine Translation , 2007, ACL.

[41]  Richard Zens,et al.  The RWTH Machine Translation System , 2006 .

[42]  Daniel Marcu,et al.  Statistical Phrase-Based Translation , 2003, NAACL.

[43]  Daniel Dominic Sleator,et al.  Parsing English with a Link Grammar , 1995, IWPT.

[44]  Arturo Trujillo.  Translation Engines: Techniques for Machine Translation , 1999 .

[45]  Hermann Ney,et al.  Novel Reordering Approaches in Phrase-Based Statistical Machine Translation , 2005, ParallelText@ACL.

[46]  Yanjun Ma,et al.  MaTrEx: the DCU machine translation system for IWSLT 2007 , 2007, IWSLT.

[47]  Andy Way,et al.  Hybrid data-driven models of machine translation , 2005, Machine Translation.

[48]  Philipp Koehn,et al.  Pharaoh: A Beam Search Decoder for Phrase-Based Statistical Machine Translation Models , 2004, AMTA.

[49]  Andy Way,et al.  Joining Hands: Developing A Sign Language Machine Translation System with and for the Deaf Community , 2007, CVHI.

[50]  Andy Way,et al.  Combining data-driven MT systems for improved sign languagetranslation , 2007 .

[51]  Andy Way,et al.  Example-based controlled translation , 2004, EAMT.

[52]  John S. White,et al.  The ARPA MT Evaluation Methodologies: Evolution, Lessons, and Future Approaches , 1994, AMTA.

[53]  Catherine N. Ball,et al.  Representation of american sign language for machine translation , 2002 .

[54]  John S. White,et al.  Evaluation of Machine Translation , 1993, HLT.

[55]  Andy Way,et al.  Labelled Dependencies in Machine Translation Evaluation , 2007, WMT@ACL.

[56]  Andy Way,et al.  Hybrid Example-Based SMT: the Best of Both Worlds? , 2005, ParallelText@ACL.

[57]  George Karypis,et al.  Evaluation of Item-Based Top-N Recommendation Algorithms , 2001, CIKM '01.

[58]  Lorraine Leeson,et al.  Moving Heads and Moving Hands: Developing a Digital Corpus of Irish Sign Language. , 2006 .

[59]  Hermann Ney,et al.  Improved backing-off for M-gram language modeling , 1995, 1995 International Conference on Acoustics, Speech, and Signal Processing.

[60]  ANDY WAY,et al.  Comparing example-based and statistical machine translation , 2005, Nat. Lang. Eng..

[61]  Barbara Celeste LeMaster.  The maintenance and loss of female and male signs in the Dublin deaf community , 1990 .

[62]  Karolina Owczarzak.  A novel dependency-based evaluation metric for machine translation , 2008 .

[63]  John Cocke,et al.  A Statistical Approach to Machine Translation , 1990, CL.

[64]  Kathryn B. Taylor,et al.  Machine Translation: From Real Users to Research , 2004, Lecture Notes in Computer Science.

[65]  S. Pinker,et al.  The Language Instinct: How the Mind Creates Language , 1994 .

[66]  S. Pinker.  The language instinct : how the mind creates language , 1995 .

[67]  Alex Waibel,et al.  The CMU statistical machine translation system , 2003, MTSUMMIT.

[68]  Éva Sáfár,et al.  The Architecture of an English-Text-to-Sign-Languages Translation System , 2001 .

[69]  Lynette van Zijl,et al.  The South African sign language machine translation project: issues on non-manual sign generation , 2006 .

[70]  Thomas Hanke.  HamNoSys – Representing Sign Language Data in Language Resources and Language Processing Contexts , 2004 .

[71]  Padraig Cunningham,et al.  Organizational Issues Arising from the Integration of the Lexicon and Concept Network in a Text Understanding System , 1991, IJCAI.

[72]  John R. Pierce,et al.  Language and Machines: Computers in Translation and Linguistics , 1966 .

[73]  Alon Lavie,et al.  METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments , 2005, IEEvaluation@ACL.

[74]  Joseph P. Turian,et al.  Evaluation of machine translation and its evaluation , 2003, MTSUMMIT.

[75]  Tony Veale,et al.  A Linguistic Approach to Sign Language Synthesis , 1994, BCS HCI.

[76]  Noam Chomsky,et al.  New Horizons in the Study of Language and Mind: Naturalism and dualism in the study of language and mind , 2008 .

[77]  Claude E. Shannon,et al.  Recent Contributions to The Mathematical Theory of Communication , 2009 .

[78]  W. Stokoe,et al.  Semiotics and Human Sign Languages , 1972 .

[79]  Andy Way,et al.  Lost in translation: the problems of using mainstream MTevaluation metrics for sign language translation , 2006 .

[80]  Lynette van Zijl,et al.  South African sign language assistive translation , 2008 .

[81]  Michael Paul,et al.  Overview of the IWSLT06 evaluation campaign , 2006, IWSLT.

[82]  John Cocke,et al.  A Statistical Approach to Language Translation , 1988, COLING.

[83]  W. Stokoe,et al.  Sign language structure: an outline of the visual communication systems of the American deaf. 1960. , 1961, Journal of deaf studies and deaf education.

[84]  Andy Way,et al.  An Example-Based Approach to Translating Sign Language , 2005, MTSUMMIT.

[85]  Saduf Naqvi.  End-user Involvement in Assistive Technology Design for the Deaf - are Artificial Forms of Sign Language Meeting the Needs of the Target Audience? , 2007, CVHI.

[86]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[87]  Vladimir I. Levenshtein,et al.  Binary codes capable of correcting deletions, insertions, and reversals , 1965 .

[88]  Jaco Fourie,et al.  The Design of a Generic Signing Avatar Animation System , 2006 .

[89]  Ian Marshall,et al.  The development of language processing support for the ViSiCAST project , 2000, Assets '00.

[90]  Ian Marshall,et al.  A Prototype Text to British Sign Language (BSL) Translation System , 2003, ACL.

[91]  J. F. Soechting,et al.  Coarticulation in Fluent Fingerspelling , 2003, The Journal of Neuroscience.

[92]  John Chandioux.  METEO : un système opérationnel pour la traduction automatique des bulletins météorologiques destinés au grand public , 1976 .

[93]  Karen Emmorey,et al.  The neural correlates of spatial language in English and American Sign Language: a PET study with hearing bilinguals , 2005, NeuroImage.

[94]  Jeremy Peckham,et al.  A new generation of spoken dialogue systems: results and lessons from the sundial project , 1993, EUROSPEECH.

[95]  Javier Macías Guarasa,et al.  Proposing a speech to gesture translation architecture for Spanish deaf people , 2008, J. Vis. Lang. Comput..

[96]  Wayne H. Ward.  Understanding spontaneous speech: the Phoenix system , 1991, [Proceedings] ICASSP 91: 1991 International Conference on Acoustics, Speech, and Signal Processing.