A Spanish speech to sign language translation system for assisting deaf-mute people

This paper describes the first experiments of a speech to sign language translation system in a real domain. The developed system is focused on the sentences spoken by an officer when assisting people in applying for, or renewing the National Identification Document (NID) and the Passport. This system translates officer explanations into sign language for deafmute people. The translation system is composed by a speech recognizer (for decoding the spoken utterance into a word sequence), a natural language translator (for converting a word sequence into a sequence of gestures belonging to the sign language), and a 3D avatar animation module (for playing the gestures). The field experiments have reported a 27.2% GER (Gesture Error Rate) and a 0.62 BLEU (BiLingual Evaluation Understudy). Index Terms: Machine Translation, Spanish Sign Language, Speech Translation, Gesture Animation

[1]  Hermann Ney,et al.  Discriminative Training and Maximum Entropy Models for Statistical Machine Translation , 2002, ACL.

[2]  N Burch-Hales,et al.  Sign language. , 1997, ASHA.

[3]  C. Penn,et al.  Sign language in South Africa: some research and clinical issues. , 1984, The South African journal of communication disorders = Die Suid-Afrikaanse tydskrif vir Kommunikasieafwykings.

[4]  Margaret Deuchar.  British Sign Language , 1984 .

[5]  Javier Macías Guarasa,et al.  Generating gestures from speech , 2004, INTERSPEECH.

[6]  Matt Huenerfauth,et al.  A Multi-Path Architecture for Machine Translation of English Text into American Sign Language Animation , 2004, NAACL.

[7]  Hermann Ney,et al.  A Systematic Comparison of Various Statistical Alignment Models , 2003, CL.

[8]  Mark Wells,et al.  Tessa, a system to aid communication with deaf people , 2002, ASSETS.

[9]  W. Stokoe,et al.  Sign language structure: an outline of the visual communication systems of the American deaf. 1960. , 1961, Journal of deaf studies and deaf education.

[10]  Andy Way,et al.  An Example-Based Approach to Translating Sign Language , 2005, MTSUMMIT.

[11]  George L. Trager,et al.  Studies in linguistics , 1942 .

[12]  Francisco Casacuberta,et al.  Machine Translation with Inferred Stochastic Finite-State Transducers , 2004, Computational Linguistics.

[13]  Jennie E. Pyers.  Indicating the body: Expression of body part terminology in American Sign Language , 2006 .

[14]  María Ángeles,et al.  Lenguaje de signos , 1992 .

[15]  Daniel Marcu,et al.  Statistical Phrase-Based Translation , 2003, NAACL.

[16]  B Woll,et al.  British sign language. , 2008, Special education: forward trends.

[17]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[18]  Fernando Fernández Martínez,et al.  New word-level and sentence-level confidence scoring using graph theory calculus and its evaluation on speech understanding , 2005, INTERSPEECH.

[19]  Hermann Ney,et al.  Statistical Sign Language Translation , 2004 .

[20]  Masako Notoya,et al.  Academic Achievement of Hearing-Impaired Children Trained by Written language-Auditory Method and Acquisition of Sign Language in Profoundly Deaf Infants , 1986 .

[21]  Taro Watanabe,et al.  A corpus-centered approach to spoken language translation , 2003, EACL.

[22]  Britta Hansen.  Varieties in Danish Sign Language and Grammatical Features of the Original Sign Language , 1975 .