Expressive ASL Recognition using Millimeter-wave Wireless Signals

Over half a million people in the United States use American Sign Language (ASL) as their primary mode of communication. Automatic ASL recognition would enable Deaf and Hard of Hearing (DHH) users to interact with others who are not familiar with ASL as well as voice-controlled digital assistants (e.g., Alexa, Siri, etc.). While ASL recognition has been extensively studied, there is a little attention given to recognition of ASL non-manual body markers. The non-manual markers are typically expressed through head, torso and shoulder movements, and add essential meaning and context to the signed sentences. In this work, we present ExASL, a sentence-level ASL recognition system using millimeter-wave radars. ExASL can recognize manual markers (hand gestures) and non-manual markers (head and torso movements). It utilizes multi-distance clustering to recognize body parts and cluster mmWave point clouds. We then present a multi-view deep learning algorithm that can learn from clustered body part representation for an expressive sentence-level recognition. Our evaluation shows that ExASL can recognize ASL sentences with a word error rate of 0.79%, sentence error rate of 1.25%, and non-manual markers with an accuracy of 83.5%.

[1]  Fei Yang,et al.  Recognizing eyebrow and periodic head gestures using CRFs for non-manual grammatical marker detection in ASL , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[2]  Panlong Yang,et al.  SignSpeaker: A Real-time, High-Precision SmartWatch-based Sign Language Translator , 2019, MobiCom.

[3]  Cecile McKee.  The Signs of Language Revisited: An Anthology to Honor Ursula Bellugi and Edward Klima (review) , 2001 .

[4]  Anant Agarwal,et al.  Sign language recognition using Microsoft Kinect , 2013, 2013 Sixth International Conference on Contemporary Computing (IC3).

[5]  Houqiang Li,et al.  Iterative Alignment Network for Continuous Sign Language Recognition , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Jie Huang,et al.  Video-based Sign Language Recognition without Temporal Segmentation , 2018, AAAI.

[7]  Shuangquan Wang,et al.  SignFi , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[8]  Ivan Poupyrev,et al.  Soli , 2016, ACM Trans. Graph..

[9]  Jürgen Schmidhuber,et al.  Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks , 2006, ICML.

[10]  Kirsti Grobel,et al.  Isolated sign language recognition using hidden Markov models , 1996, 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation.

[11]  Parth H. Pathak,et al.  Monitoring vital signs using millimeter wave , 2016, MobiHoc.

[12]  J. Feldman,et al.  The severely to profoundly hearing-impaired population in the United States: prevalence estimates and demographics. , 2001, Journal of the American Academy of Audiology.

[13]  Fei Yang,et al.  Recognition of Nonmanual Markers in American Sign Language (ASL) Using Non-Parametric Adaptive 2D-3D Face Tracking , 2012, LREC.