Including Signed Languages in Natural Language Processing

Signed languages are the primary means of communication for many deaf and hard of hearing individuals. Since signed languages exhibit all the fundamental linguistic properties of natural language, we believe that tools and theories of Natural Language Processing (NLP) are crucial towards its modeling. However, existing research in Sign Language Processing (SLP) seldom attempt to explore and leverage the linguistic organization of signed languages. This position paper calls on the NLP community to include signed languages as a research area with high social and scientific impact. We first discuss the linguistic properties of signed languages to consider during their modeling. Then, we review the limitations of current SLP models and identify the open challenges to extend NLP to signed languages. Finally, we urge (1) the adoption of an efficient tokenization method; (2) the development of linguistically-informed models; (3) the collection of real-world signed language data; (4) the inclusion of local signed language communities as an active and leading voice in the direction of research.

[1]  King-Sun Fu,et al.  IEEE Transactions on Pattern Analysis and Machine Intelligence Publication Information , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Yoav Goldberg,et al.  Data Augmentation for Sign Language Gloss Translation , 2021, MTSUMMIT.

[3]  Sergio Escalera,et al.  Sign Language Recognition: A Deep Survey , 2021, Expert Syst. Appl..

[4]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Steven Bird,et al.  Decolonising Speech and Language Technology , 2020, COLING.

[6]  Ben Saunders,et al.  Everybody Sign Now: Translating Spoken Language to Photo Realistic Sign Language Video , 2020, ArXiv.

[7]  Anara Sandygulova,et al.  A Dataset for Linguistic Understanding, Visual Evaluation, and Recognition of Sign Languages: The K-RSL , 2020, CONLL.

[8]  Gabriel Synnaeve,et al.  MLS: A Large-Scale Multilingual Dataset for Speech Research , 2020, INTERSPEECH.

[9]  Oscar Koller,et al.  Multi-channel Transformers for Multi-articulatory Sign Language Translation , 2020, ECCV Workshops.

[10]  Oscar Koller,et al.  Quantitative Survey of the State of the Art in Sign Language Recognition , 2020, ArXiv.

[11]  Hacer Yalim Keles,et al.  AUTSL: A Large Scale Multi-Modal Turkish Sign Language Dataset and Baseline Methods , 2020, IEEE Access.

[12]  Petros Daras,et al.  A Comprehensive Study on Sign Language Recognition Methods , 2020, ArXiv.

[13]  Marc Schulder,et al.  Extending the Public DGS Corpus in Size and Depth , 2020, SIGNLANG.

[14]  Amy Isard,et al.  Approaches to the Anonymisation of Sign Language Corpora , 2020, SIGNLANG.

[15]  Saad Hassan,et al.  An Isolated-Signing RGBD Dataset of 100 American Sign Language Signs Produced by Fluent ASL Signers , 2020, SIGNLANG.

[16]  Necati Cihan Camgöz,et al.  Progressive Transformers for End-to-End Sign Language Production , 2020, ECCV.

[17]  Jesse Read,et al.  Better Sign Language Translation with STMC-Transformer , 2020, COLING.

[18]  Oscar Koller,et al.  Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Jan Zelinka,et al.  Neural Sign Language Synthesis: Words Are Our Glosses , 2020, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).

[20]  Joseph J. Murray,et al.  The Importance of Signed Languages for Deaf Children and Their Families , 2020 .

[21]  Qinkun Xiao,et al.  Skeleton-based Chinese sign language recognition and generation for bidirectional communication between deaf and hearing people , 2020, Neural Networks.

[22]  Necati Cihan Camgoz,et al.  Text2Sign: Towards Sign Language Production Using Neural Machine Translation and Generative Adversarial Networks , 2020, International Journal of Computer Vision.

[23]  Hongdong Li,et al.  Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison , 2019, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).

[24]  Jesse Read,et al.  Attention is All You Sign: Sign Language Translation with Transformers , 2020 .

[25]  Michèle Gouiffès,et al.  Automatic Segmentation of Sign Language into Subtitle-Units , 2020, ECCV Workshops.

[26]  Karen Livescu,et al.  Fingerspelling Recognition in the Wild With Iterative Visual Attention , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[27]  Meredith Ringel Morris,et al.  Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective , 2019, ASSETS.

[28]  Ankur Bapna,et al.  Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges , 2019, ArXiv.

[29]  Kenneth P. Camilleri,et al.  Sign Language Detection “in the Wild” with Recurrent Neural Networks , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[30]  Heike Brock,et al.  Learning Motion Disfluencies for Automatic Sign Language Segmentation , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[31]  Oscar Koller,et al.  MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language , 2018, BMVC.

[32]  Sang-Ki Ko,et al.  Neural Sign Language Translation based on Human Keypoint Estimation , 2018, Applied Sciences.

[33]  Louis-Philippe Morency,et al.  Multimodal Machine Learning: A Survey and Taxonomy , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  T. Richardson.  Language Deprivation and Deaf Mental Health , 2018 .

[35]  Gregory Shakhnarovich,et al.  American Sign Language Fingerspelling Recognition in the Wild , 2018, 2018 IEEE Spoken Language Technology Workshop (SLT).

[36]  Richard Bowden,et al.  Sign Language Production using Neural Machine Translation and Generative Adversarial Networks , 2018, BMVC.

[37]  Jordan Fenlon,et al.  Indicating verbs as typologically unique constructions: Reconsidering verb ‘agreement’ in sign languages , 2018, Glossa: a journal of general linguistics.

[38]  Diane Brentari,et al.  Sign Language Phonology , 2018, Oxford Research Encyclopedia of Linguistics.

[39]  Hermann Ney,et al.  Neural Sign Language Translation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[40]  Doug Downey,et al.  Construction of the Literature Graph in Semantic Scholar , 2018, NAACL.

[41]  Sarah Ebling,et al.  SMILE Swiss German Sign Language Dataset , 2018, LREC.

[42]  Diane Brentari,et al.  Emphatic fingerspelling as code-mixing in American Sign Language , 2018 .

[43]  A. Schembri,et al.  Modification of indicating verbs in British Sign Language: A corpus-based study , 2018 .

[44]  Iasonas Kokkinos,et al.  DensePose: Dense Human Pose Estimation in the Wild , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[45]  Jie Huang,et al.  Video-based Sign Language Recognition without Temporal Segmentation , 2018, AAAI.

[46]  Maity Gouranga,et al.  COMPREHENSIVE STUDY OF , 2018 .

[47]  Changshui Zhang,et al.  Recurrent Convolutional Neural Networks for Continuous Sign Language Recognition by Staged Optimization , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Annelies Kusters,et al.  Innovations in deaf studies: The role of deaf scholars , 2017 .

[49]  Xiu-Shen Wei,et al.  Adversarial PoseNet: A Structure-Aware Convolutional Network for Human Pose Estimation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[50]  L. Levin,et al.  Language deprivation syndrome: a possible neurodevelopmental disorder with sociocultural origins , 2017, Social Psychiatry and Psychiatric Epidemiology.

[51]  Frank M. Shipman,et al.  Detecting and Identifying Sign Languages through Visual Features , 2016, 2016 IEEE International Symposium on Multimedia (ISM).

[52]  T. Humphries,et al.  Avoiding Linguistic Neglect of Deaf Children , 2016, Social Service Review.

[53]  C. Padden.  Interaction of Morphology and Syntax in American Sign Language , 2016 .

[54]  Markus Steinbach,et al.  A DRT Analysis of Discourse Referents and Anaphora Resolution in Sign Language , 2016, J. Semant..

[55]  Thomas Hanke,et al.  Using a Language Technology Infrastructure for German in order to Anonymize German Sign Language Corpus Data , 2016, LREC.

[56]  Manuel Carreiras,et al.  LSE-Sign: A lexical database for Spanish Sign Language , 2015, Behavior Research Methods.

[57]  Jordan Fenlon,et al.  Building BSL SignBank: The Lemma Dilemma Revisited , 2015 .

[58]  Johanna Mesch,et al.  Gloss annotations in the Swedish Sign Language Corpus , 2015 .

[59]  Vadim Kimmelman,et al.  Information structure in Russian Sign Language and Sign Language of the Netherlands , 2015 .

[60]  K. Cormier,et al.  Rethinking constructed action , 2015 .

[61]  Hermann Ney,et al.  Extensions of the Sign Language Recognition and Translation Corpus RWTH-PHOENIX-Weather , 2014, LREC.

[62]  Peter Wittenburg,et al.  Automatic sign language identification , 2013, 2013 IEEE International Conference on Image Processing.

[63]  Trevor Johnston,et al.  Auslan Corpus Annotation Guidelines , 2013 .

[64]  Bernt Schiele,et al.  Articulated people detection and pose estimation: Reshaping the future , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[65]  Onno Crasborn,et al.  OF SENTENCES IN SIGNED LANGUAGES , 2022 .

[66]  Wendy Sandler,et al.  The Phonological Organization of Sign Languages , 2012, Lang. Linguistics Compass.

[67]  Johanna Mesch,et al.  From meaning to signs and back:Lexicography and the Swedish Sign Language Corpus , 2012, LREC 2012.

[68]  Cynthia B. Roy,et al.  Discourse in Signed Languages , 2011 .

[69]  Scott K. Liddell,et al.  Toward a Phonetic Representation of Signs: Sequentiality and Contrast , 2011 .

[70]  Gaurav Mathur,et al.  A featural approach to verb agreement in signed languages , 2011 .

[71]  Wendy Sandler,et al.  Prosody and Syntax in Sign Languages. , 2010, Transactions of the Philological Society. Philological Society.

[72]  Manuel Carreiras,et al.  Sign Language Processing , 2010, Lang. Linguistics Compass.

[73]  Lale Akarun,et al.  Automatic sign segmentation from continuous signing via multiple sequence alignment , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[74]  D. Mertens,et al.  Research Ethics in Sign Language Communities , 2009 .

[75]  Trevor Johnston,et al.  From archive to corpus: transcription and annotation in the creation of signed language corpora , 2008, PACLIC.

[76]  Onno Crasborn,et al.  The Corpus NGT: An online corpus for professionals and laymen , 2008 .

[77]  Adam Schembri,et al.  Australian Sign Language: Auslan: An Introduction to Sign Language Linguistics , 2007 .

[78]  Wendy Sandler,et al.  Sign Language and Linguistic Universals: Entering the lexicon: lexicalization, backformation, and cross-modal borrowing , 2006 .

[79]  B. Parton.  Sign language recognition and translation: a multidisciplined approach from the field of artificial intelligence. , 2005, Journal of deaf studies and deaf education.

[80]  H. Ney,et al.  Modeling Image Variability in Appearance-Based Gesture Recognition , 2006 .

[81]  Surendra Ranganath,et al.  Automatic Sign Language Analysis: A Survey and the Future beyond Lexical Meaning , 2005, IEEE Trans. Pattern Anal. Mach. Intell..

[82]  W. Stokoe.  Sign language structure: an outline of the visual communication systems of the American deaf. 1960. , 1961, Journal of deaf studies and deaf education.

[83]  S. Wilcox,et al.  Rethinking Classifiers. Emmorey, K. (Ed.). (2003). Perspectives on Classifier Constructions in Sign Languages. Mahwah, NJ: Lawrence Erlbaum Associates. 332 pages. Hardcover. $69.95. , 2004 .

[84]  Javier Ramírez,et al.  Efficient voice activity detection algorithms using long-term speech information , 2004, Speech Commun..

[85]  Paul G. Dudis.  Body partitioning and real-space blends , 2004 .

[86]  Rethinking Classifiers , 2004 .

[87]  Scott K. Liddell.  Grammar, Gesture, and Meaning in American Sign Language , 2003 .

[88]  T. Johnston,et al.  On defining lexeme in a signed language , 1999 .

[89]  Wonyong Sung,et al.  A statistical model-based voice activity detection , 1999, IEEE Signal Processing Letters.

[90]  Melanie Metzger,et al.  Gesture in sign language discourse , 1998 .

[91]  Carol Padden,et al.  The ASL lexicon , 1998 .

[92]  D L Jaffe.  Evolution of mechanical fingerspelling hands for people who are deaf-blind. , 1994, Journal of rehabilitation research and development.

[93]  Sherman Wilcox,et al.  The phonetics of fingerspelling , 1992 .

[94]  Scott K. Liddell,et al.  American Sign Language: The Phonological Base , 2013 .

[95]  Ted Suppalla,et al.  The Classifier System in American Sign Language , 1986 .

[96]  Lucinda Ferreira Brito.  Similarities & Differences in Two Brazilian Sign Languages , 2013 .

[97]  S. Shroyer,et al.  Signs Across America: A Look at Regional Differences in American Sign Language , 1984 .

[98]  R. Battison,et al.  Lexical Borrowing in American Sign Language , 1978 .

[99]  U. Bellugi,et al.  A comparison of sign language and spoken language , 1972 .

[100]  Jim Kakumasu.  Urubú Sign Language , 1968, International Journal of American Linguistics.