WLASL-LEX: a Dataset for Recognising Phonological Properties in American Sign Language

Signed Language Processing (SLP) concerns the automated processing of signed languages, the main means of communication of Deaf and hearing impaired individuals. SLP features many different tasks, ranging from sign recognition to translation and production of signed speech, but has been overlooked by the NLP community thus far.In this paper, we bring to attention the task of modelling the phonology of sign languages. We leverage existing resources to construct a large-scale dataset of American Sign Language signs annotated with six different phonological properties. We then conduct an extensive empirical study to investigate whether data-driven end-to-end and feature-based approaches can be optimised to automatically recognise these properties. We find that, despite the inherent challenges of the task, graph-based neural networks that operate over skeleton features extracted from raw videos are able to succeed at the task to a varying degree. Most importantly, we show that this performance pertains even on signs unobserved during training.

[1]  Yun Fu,et al.  Sign Language Recognition via Skeleton-Aware Multi-Model Ensemble , 2021, ArXiv.

[2]  Angelo Cangelosi,et al.  Phonology Recognition in American Sign Language , 2021, ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[3]  Takaaki Shiratori,et al.  FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integration , 2021, 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW).

[4]  Malihe Alikhani,et al.  Including Signed Languages in Natural Language Processing , 2021, ACL.

[5]  Yun Fu,et al.  Skeleton Aware Multi-modal Sign Language Recognition , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[6]  Sergio Escalera,et al.  Sign Language Recognition: A Deep Survey , 2021, Expert Syst. Appl..

[7]  Oscar Koller,et al.  Quantitative Survey of the State of the Art in Sign Language Recognition , 2020, ArXiv.

[8]  Ioannis Tsochantaridis,et al.  Real-Time Sign Language Detection using Human Pose Estimation , 2020, ECCV Workshops.

[9]  Marzieh Razavi,et al.  Towards Multilingual Sign Language Recognition , 2020, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[10]  Goran Nenadic,et al.  A Framework for Evaluation of Machine Reading Comprehension Gold Standards , 2020, LREC.

[11]  D. Brentari.  Sign Language Phonology , 2019 .

[12]  Hongdong Li,et al.  Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison , 2019, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).

[13]  Yang Zhao,et al.  Deep High-Resolution Representation Learning for Visual Recognition , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Mei-Ling Shyu,et al.  A Survey on Deep Learning , 2018, ACM Comput. Surv..

[15]  Hermann Ney,et al.  Neural Sign Language Translation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[16]  Armir Bujari,et al.  GHio-Ca: An Android Application for Automatic Image Classification , 2017, GOODTECHS.

[17]  Lihong Zheng,et al.  Recent Advances of Deep Learning for Sign Language Recognition , 2017, 2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA).

[18]  Andrew Zisserman,et al.  Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Sergio Escalera,et al.  A Survey on Deep Learning Based Approaches for Action and Gesture Recognition in Image Sequences , 2017, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).

[20]  Karen Emmorey,et al.  ASL-LEX: A lexical database of American Sign Language , 2017, Behavior research methods.

[21]  Danqi Chen,et al.  A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task , 2016, ACL.

[22]  Scott K. Liddell,et al.  American Sign Language: The Phonological Base , 2013 .

[23]  Peter Wittenburg,et al.  Automatic sign language identification , 2013, 2013 IEEE International Conference on Image Processing.

[24]  Wendy Sandler,et al.  The Phonological Organization of Sign Languages , 2012, Lang. Linguistics Compass.

[25]  R. Battison,et al.  Lexical Borrowing in American Sign Language , 1978 .

[26]  B. Matthews.  Comparison of the predicted and observed secondary structure of T4 phage lysozyme. , 1975, Biochimica et biophysica acta.

[27]  Sandrine Tornay,et al.  Explainable Phonology-based Approach for Sign Language Recognition and Assessment , 2021 .

[28]  Kenneth P. Camilleri,et al.  Phonologically-Meaningful Subunits for Deep Learning-Based Sign Language Recognition , 2020, ECCV Workshops.

[29]  Dimitris N. Metaxas,et al.  Scalable ASL Sign Recognition using Model-based Machine Learning and Linguistically Annotated Corpora , 2022 .