AT 4 SSL Proceedings of the Second International Workshop on Automatic Translation for Signed and Spoken Languages 15 June 2023

Sign language processing is the field of research that aims to recognize, retrieve, and spot signs in videos. Various approaches have been developed, varying in whether they use linguistic features and whether they use landmark detection tools or not. Incorporating linguistics holds promise for improving sign language processing in terms of performance, generalizability, and explainability. This paper focuses on the task of sign spotting and aims to expand on the approximative linguistic features that have been used in previous work, and to understand when linguistic features deliver an improvement over landmark features. We detect landmarks with Mediapipe and extract linguistically relevant features from them, including handshape, orientation, location, and movement. We compare a sign spotting model using linguistic features with a model operating on landmarks directly, finding that the approximate linguistic features tested in this paper capture some aspects of signs better than the landmark features, while they are worse for others.

[1]  Rico Sennrich,et al.  SLTUNET: A Simple Unified Model for Sign Language Translation , 2023, ICLR.

[2]  Sarah Ebling,et al.  Considerations for meaningful sign language machine translation based on glosses , 2022, ACL.

[3]  Ohad Fried,et al.  Ham2Pose: Animating Sign Language Notation into Pose Sequences , 2022, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Adrián Núñez-Marcos,et al.  A survey on Sign Language machine translation , 2022, Expert Syst. Appl..

[5]  Bencie Woll,et al.  BBC-Oxford British Sign Language Dataset , 2021, ArXiv.

[6]  Thomas Hanke,et al.  Overview of Datasets for the Sign Languages of Europe , 2021 .

[7]  Ben Saunders,et al.  Anonysign: Novel Human Appearance Synthesis for Sign Language Video Anonymisation , 2021, 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021).

[8]  Richard E. Ladner,et al.  The FATE Landscape of Sign Language AI Datasets , 2021, ACM Trans. Access. Comput..

[9]  Wengang Zhou,et al.  Improving Sign Language Translation with Monolingual Data by Sign Back-Translation , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Yoav Goldberg,et al.  Data Augmentation for Sign Language Gloss Translation , 2021, MTSUMMIT.

[11]  Malihe Alikhani,et al.  Including Signed Languages in Natural Language Processing , 2021, ACL.

[12]  Xavier Giró-i-Nieto,et al.  Can Everybody Sign Now? Exploring Sign Language Video Generation from 2D Poses , 2020, ArXiv.

[13]  Ben Saunders,et al.  Everybody Sign Now: Translating Spoken Language to Photo Realistic Sign Language Video , 2020, ArXiv.

[14]  Jörg Tiedemann,et al.  OPUS-MT – Building open translation services for the World , 2020, EAMT.

[15]  Thomas Hanke,et al.  Public DGS Corpus: Annotation Conventions / Öffentliches DGS-Korpus: Annotationskonventionen , 2020 .

[16]  Joon Son Chung,et al.  BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues , 2020, ECCV.

[17]  Marc Schulder,et al.  Extending the Public DGS Corpus in Size and Depth , 2020, SIGNLANG.

[18]  Jesse Read,et al.  Better Sign Language Translation with STMC-Transformer , 2020, COLING.

[19]  Oscar Koller,et al.  Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Marjan Ghazvininejad,et al.  Multilingual Denoising Pre-training for Neural Machine Translation , 2020, Transactions of the Association for Computational Linguistics.

[21]  Necati Cihan Camgoz,et al.  Text2Sign: Towards Sign Language Production Using Neural Machine Translation and Generative Adversarial Networks , 2020, International Journal of Computer Vision.

[22]  Parteek Kumar,et al.  Sign Language Recognition Systems: A Decade Systematic Literature Review , 2019, Archives of Computational Methods in Engineering.

[23]  Meredith Ringel Morris,et al.  Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective , 2019, ASSETS.

[24]  Rico Sennrich,et al.  Revisiting Low-Resource Neural Machine Translation: A Case Study , 2019, ACL.

[25]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Richard Bowden,et al.  Sign Language Production using Neural Machine Translation and Generative Adversarial Networks , 2018, BMVC.

[27]  Alexei A. Efros,et al.  Everybody Dance Now , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[28]  Jan Kautz,et al.  Video-to-Video Synthesis , 2018, NeurIPS.

[29]  Robin L. Thompson,et al.  Iconicity in Signed and Spoken Vocabulary: A Comparison Between American Sign Language, British Sign Language, English, and Spanish , 2018, Front. Psychol..

[30]  Hermann Ney,et al.  Neural Sign Language Translation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[31]  Taku Kudo,et al.  Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates , 2018, ACL.

[32]  Matt Post,et al.  A Call for Clarity in Reporting BLEU Scores , 2018, WMT.

[33]  R. Stamp,et al.  Variation and Change in Varieties of British Sign Language in England , 2018 .

[34]  Iasonas Kokkinos,et al.  DensePose: Dense Human Pose Estimation in the Wild , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[35]  Florian Schiel,et al.  Multilingual processing of speech via web services , 2017, Comput. Speech Lang..

[36]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[37]  Annelies Kusters,et al.  Innovations in deaf studies: Critically mapping the Field , 2017 .

[38]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Martin Wattenberg,et al.  Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation , 2016, TACL.

[40]  Lior Wolf,et al.  Using the Output Embedding to Improve Language Models , 2016, EACL.

[41]  Maja Popovic,et al.  chrF deconstructed: beta parameters and n-gram weights , 2016, WMT.

[42]  Erik Schultes,et al.  The FAIR Guiding Principles for scientific data management and stewardship , 2016, Scientific Data.

[43]  Jemina Napier,et al.  Sign Language in Action , 2016 .

[44]  Inam Ismael Taher.  The Problematic Forms of Nominalization in English: Gerund, Verbal Noun, and Deverbal Noun , 2015 .

[45]  Jordan Fenlon,et al.  Building the British Sign Language Corpus , 2013 .

[46]  Peter Wittenburg,et al.  ELAN: a Professional Framework for Multimodality Research , 2006, LREC.

[47]  Myriam Vermeerbergen,et al.  Past and current trends in sign language research , 2006 .

[48]  Hennie Brugman,et al.  Collaborative Annotation of Sign Language Data with Peer-to-Peer Technology , 2004, LREC.

[49]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[50]  Norman I. Badler,et al.  A machine translation system from English to American Sign Language , 2000, AMTA.

[51]  Bencie Woll,et al.  The Linguistics of British Sign Language: An Introduction , 1999 .

[52]  Aravind K. Joshi,et al.  Using Lexicalized Tags for Machine Translation , 1990, COLING.

[53]  Stuart M. Shieber,et al.  Synchronous Tree-Adjoining Grammars , 1990, COLING.

[54]  A. Savitzky,et al.  Smoothing and Differentiation of Data by Simplified Least Squares Procedures. , 1964 .

[55]  Hope E. Morgan,et al.  Facilitating the Spread of New Sign Language Technologies across Europe , 2022, SIGNLANG.

[56]  Mathieu De Coster,et al.  Sign Language Translation: Ongoing Development, Challenges and Innovations in the SignON Project , 2022, EAMT.

[57]  J. Blat,et al.  Sign Language Machine Translation and the Sign Language Lexicon: A Linguistically Informed Approach , 2022, AMTA.

[58]  Santiago Egea Gómez,et al.  Challenges with Sign Language Datasets for Sign Language Recognition and Translation , 2022, LREC.

[59]  Horacio Saggion,et al.  Syntax-aware Transformers for Neural Machine Translation: The Case of Text to Sign Gloss Translation , 2021, BUCC.

[60]  Irene Murtagh.  A Linguistically Motivated Computational Framework for Irish Sign Language , 2019 .

[61]  J. Kristoffersen,et al.  Improving Lemmatisation Consistency without a Phonological Description The Danish Sign Language Corpus and Dictionary Project , 2018 .

[62]  Robert Bayley,et al.  Quantitative sociolinguistics and sign languages: Implications for sociolinguistic theory , 2016 .

[63]  Florian Schiel,et al.  A statistical model for predicting pronunciation , 2015, ICPhS.

[64]  Mohamed Jemni,et al.  English-ASL Gloss Parallel Corpus 2012: ASLG-PC12 , 2012 .

[65]  Lorraine Leeson,et al.  Irish sign language : a cognitive linguistic account , 2012 .

[66]  Lorraine Leeson,et al.  Moving Heads and Moving Hands: Developing a Digital Corpus of Irish Sign Language. , 2006 .

[67]  H. Hughes.  The Cambridge Grammar of the English Language , 2003 .

[68]  Piek Vossen,et al.  EuroWordNet: general document , 2002 .

[69]  飯島 周.  「会話の文法」に関する一考察 : Longman Grammar of Spoken and Written Englishの場合 , 1999 .