Towards Multilingual Sign Language Recognition

Sign language recognition involves modeling of multichannel information such as, hand shapes, hand movements. This requires also sufficient sign language specific data. This is a challenge as sign languages are inherently under-resourced. In the literature, it has been shown that hand shape information can be estimated by pooling resources from multiple sign languages. Such a capability does not exist yet for modeling hand movement information. In this paper, we develop a multilingual sign language approach, where hand movement modeling is also done with target sign language independent data by derivation of hand movement subunits. We validate the proposed approach through an investigation on Swiss German Sign Language, German Sign Language and Turkish Sign Language, and demonstrate that sign language recognition systems can be effectively developed by using multilingual sign language resources.

[1]  Alex Pentland,et al.  Real-Time American Sign Language Recognition Using Desk and Wearable Computer Based Video , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  George Awad,et al.  Boosted subunits: a framework for recognising sign language from videos , 2013, IET Image Process..

[3]  Dimitris N. Metaxas,et al.  Parallel hidden Markov models for American sign language recognition , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[4]  Wen Gao,et al.  A novel approach to automatically extracting basic units from Chinese sign language , 2004, ICPR 2004.

[5]  Nicolas Pugeault,et al.  Sign language recognition using sub-units , 2012, J. Mach. Learn. Res..

[6]  Marzieh Razavi,et al.  On modeling context-dependent clustered states: Comparing HMM/GMM, hybrid HMM/ANN and KL-HMM approaches , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[7]  Mathew Magimai.-Doss,et al.  Subunits Inference and Lexicon Development Based on Pairwise Comparison of Utterances and Signs , 2019, Inf..

[8]  Nicolas Pugeault,et al.  Sign Language Recognition using Sequential Pattern Trees , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[9]  Elakkiya R,et al.  Extricating Manual and Non-Manual Features for Subunit Level Medical Sign Modelling in Automatic Sign Language Classification and Recognition , 2017, Journal of medical systems.

[10]  Richard Bowden,et al.  HMM-based Approaches to Model Multichannel Information in Sign Language Inspired from Articulatory Features-based Speech Processing , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[11]  Jithendra Vepa,et al.  An Acoustic Model Based on Kullback-Leibler Divergence for Posterior Features , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[12]  Wen Gao,et al.  A novel approach to automatically extracting basic units from Chinese sign language , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[13]  Hermann Ney,et al.  May the force be with you: Force-aligned signwriting for automatic subunit annotation of corpora , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[14]  Roberto Cipolla,et al.  Real-time Interpretation of Hand Motions using a Sparse Bayesian Classifier on Motion Gradient Orientation Images , 2005, BMVC.

[15]  Karl-Friedrich Kraiss,et al.  Towards an Automatic Sign Language Recognition System Using Subunits , 2001, Gesture Workshop.

[16]  Lale Akarun,et al.  Sign Language Recognition for Assisting the Deaf in Hospitals , 2016, HBU.

[17]  Ramya Rasipuram,et al.  Acoustic and lexical resource constrained ASR using language-independent acoustic model and language-dependent probabilistic lexical model , 2015, Speech Commun..

[18]  Tadashi Kitamura,et al.  Subunit Modeling for Japanese Sign Language Recognition Based on Phonetically Depend Multi-stream Hidden Markov Models , 2013, HCI.

[19]  Ramya Rasipuram,et al.  Articulatory feature based continuous speech recognition using probabilistic lexical modeling , 2016, Comput. Speech Lang..

[20]  Oscar Koller,et al.  SubUNets: End-to-End Hand Shape and Continuous Sign Language Recognition , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[21]  Hermann Ney,et al.  Deep Sign: Hybrid CNN-HMM for Continuous Sign Language Recognition , 2016, BMVC.

[22]  Sarah Ebling,et al.  SMILE Swiss German Sign Language Dataset , 2018, LREC.

[23]  Bencie Woll,et al.  The Linguistics of British Sign Language: An Introduction , 1999 .

[24]  Thomas Hanke.  HamNoSys – Representing Sign Language Data in Language Resources and Language Processing Contexts , 2004 .

[25]  Hermann Ney,et al.  Weakly Supervised Learning with Multi-Stream CNN-LSTM-HMMs to Discover Sequential Parallelism in Sign Language Videos , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Petros Maragos,et al.  Advances in phonetics-based sub-unit modeling for transcription alignment and sign language recognition , 2011, CVPR 2011 WORKSHOPS.

[27]  Richard Bowden,et al.  Sign Language Recognition , 2011, Visual Analysis of Humans.

[28]  R. A. Leibler,et al.  On Information and Sufficiency , 1951 .

[29]  Petros Maragos,et al.  Model-level data-driven sub-units for signs in videos of continuous Sign Language , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[30]  Hermann Ney,et al.  Deep Hand: How to Train a CNN on 1 Million Hand Images When Your Data is Continuous and Weakly Labelled , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Hervé Bourlard,et al.  Using KL-based acoustic models in a large vocabulary recognition task , 2008, INTERSPEECH.

[32]  George Awad,et al.  Novel boosting framework for subunit-based sign language recognition , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[33]  George Awad,et al.  Modelling and segmenting subunits for sign language recognition based on hand motion analysis , 2009, Pattern Recognit. Lett..