Syntax-aware Transformers for Neural Machine Translation: The Case of Text to Sign Gloss Translation

It is well-established that the preferred mode of communication of the deaf and hard of hearing (DHH) community are Sign Languages (SLs), but they are considered low resource languages where natural language processing technologies are of concern. In this paper we study the problem of text to SL gloss Machine Translation (MT) using Transformer-based architectures. Despite the significant advances of MT for spoken languages in the recent couple of decades, MT is in its infancy when it comes to SLs. We enrich a Transformer-based architecture aggregating syntactic information extracted from a dependency parser to word-embeddings. We test our model on a well-known dataset showing that the syntax-aware model obtains performance gains in terms of MT evaluation metrics.

[1]  Luísa Coheur,et al.  From European Portuguese to Portuguese Sign Language , 2015, SLPAT@Interspeech.

[2]  Oscar Koller,et al.  Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Jesse Read,et al.  Better Sign Language Translation with STMC-Transformer , 2020, COLING.

[4]  Marjan Ghazvininejad,et al.  Multilingual Denoising Pre-training for Neural Machine Translation , 2020, Transactions of the Association for Computational Linguistics.

[5]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[6]  Veronica Lopez Ludeña,et al.  Language Resources for Spanish - Spanish Sign Language (LSE) translation , 2010, LREC 2010.

[7]  Rico Sennrich,et al.  Linguistic Input Features Improve Neural Machine Translation , 2016, WMT.

[8]  Thomas Hanke.  HamNoSys – Representing Sign Language Data in Language Resources and Language Processing Contexts , 2004 .

[9]  Maximilian Wendt,et al.  HDT-UD: A very large Universal Dependencies Treebank for German , 2019, Proceedings of the Third Workshop on Universal Dependencies (UDW, SyntaxFest 2019).

[10]  Omer Levy,et al.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension , 2019, ACL.

[11]  Necati Cihan Camgoz,et al.  Text2Sign: Towards Sign Language Production Using Neural Machine Translation and Generative Adversarial Networks , 2020, International Journal of Computer Vision.

[12]  Lijun Wu,et al.  Achieving Human Parity on Automatic Chinese to English News Translation , 2018, ArXiv.

[13]  Carmen García-Mateo,et al.  CORILSE: a Spanish Sign Language Repository for Linguistic Analysis , 2016, LREC.

[14]  Anara Sandygulova,et al.  Evaluation of Manual and Non-manual Components for Sign Language Recognition , 2020, LREC.

[15]  Yoav Goldberg,et al.  Data Augmentation for Sign Language Gloss Translation , 2021, MTSUMMIT.

[16]  Taku Kudo,et al.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing , 2018, EMNLP.

[17]  Tommi Jantunen,et al.  Is There Any Hope for Developing Automated Translation Technology for Sign Languages? , 2021 .

[18]  Chin-Yew Lin,et al.  ROUGE: A Package for Automatic Evaluation of Summaries , 2004, ACL 2004.

[19]  Andrew Y. Ng,et al.  Parsing with Compositional Vector Grammars , 2013, ACL.

[20]  Hermann Ney,et al.  Neural Sign Language Translation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[21]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[22]  Fabrizio Nunnari,et al.  A Data Augmentation Approach for Sign-Language-To-Text Translation In-The-Wild , 2021, LDK.

[23]  Juan Carlos López,et al.  Translating bus information into sign language for deaf people , 2014, Eng. Appl. Artif. Intell..

[24]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[25]  Larwan Berke,et al.  Methods for Evaluation of Imperfect Captioning Tools by Deaf or Hard-of-Hearing Users at Different Reading Literacy Levels , 2018, CHI.

[26]  Wolfgang Lezius,et al.  TIGER: Linguistic Interpretation of a German Corpus , 2004 .

[27]  Matt Post,et al.  A Call for Clarity in Reporting BLEU Scores , 2018, WMT.

[28]  Miles Osborne,et al.  Statistical Machine Translation , 2010, Encyclopedia of Machine Learning and Data Mining.

[29]  Alon Lavie,et al.  METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments , 2005, IEEvaluation@ACL.

[30]  Margriet Verlinden,et al.  SYNTHETIC SIGNING FOR THE DEAF : eSIGN , 2005 .

[31]  Colin Raffel,et al.  mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer , 2021, NAACL.

[32]  Matthew G. Snover,et al.  A Study of Translation Edit Rate with Targeted Human Annotation , 2006, AMTA.

[33]  José Colás Pasamontes,et al.  A rule-based translation from written Spanish to Spanish Sign Language glosses , 2014, Comput. Speech Lang..

[34]  Martin Wattenberg,et al.  Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation , 2016, TACL.