Real-time Embedded Recognition of Sign Language Alphabet Fingerspelling in an IMU-Based Glove

Data gloves have numerous applications, including enabling novel human-computer interaction and automated recognition of large sets of gestures, such as those used for sign language. For most of these applications, it is important to build mobile and self-contained applications that run without the need for frequent communication with additional services on a back-end server. We present in this paper a data glove prototype, based on multiple small Inertial Measurement Units (IMUs), with a glove-embedded classifier for the french sign language. In an extensive set of experiments with 57 participants, our system was tested by repeatedly fingerspelling the French Sign Language (LSF) alphabet. Results show that our system is capable of detecting the LSF alphabet with a mean accuracy score of 92% and an F1 score of 91%, with all detections performed on the glove within 63 milliseconds.

[1]  Waqar Ahmed,et al.  Cost effective portable system for sign language gesture recognition , 2008, 2008 IEEE International Conference on System of Systems Engineering.

[2]  Surapa Thiemjarus,et al.  Design of an Assistive Communication Glove Using Combined Sensory Channels , 2012, 2012 Ninth International Conference on Wearable and Implantable Body Sensor Networks.

[3]  Kristofer S. J. Pister,et al.  Acceleration sensing glove (ASG) , 1999, Digest of Papers. Third International Symposium on Wearable Computers.

[4]  Khaled Assaleh,et al.  Glove-Based Continuous Arabic Sign Language Recognition in User-Dependent Mode , 2015, IEEE Transactions on Human-Machine Systems.

[5]  Juan M. Bogado,et al.  GLOVE-BASED GESTURE RECOGNITION SYSTEM , 2012 .

[6]  Kristofer S. J. Pister,et al.  Acceleration Sensing Glove , 1999 .

[7]  Yu Huang,et al.  A Concept Grounding Approach for Glove-Based Gesture Recognition , 2011, 2011 Seventh International Conference on Intelligent Environments.

[8]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[9]  David Zeltzer,et al.  A survey of glove-based input , 1994, IEEE Computer Graphics and Applications.

[10]  Ming C. Leu,et al.  Linguistic properties based on American Sign Language isolated word recognition with artificial neural networks using a sensory glove and motion tracker , 2007, Neurocomputing.

[11]  Jeongsoo Lee,et al.  Development of a finger motion measurement system using linear potentiometers , 2014, 2014 IEEE/ASME International Conference on Advanced Intelligent Mechatronics.

[12]  Subhas Mukhopadhyay,et al.  MEMS based IMU for tilting measurement: Comparison of complementary and kalman filter based data fusion , 2015, 2015 IEEE 10th Conference on Industrial Electronics and Applications (ICIEA).

[13]  Tomohiro Kuroda,et al.  Consumer price data-glove for sign language recognition , 2004 .

[14]  Johannes Wagner,et al.  Bi-channel sensor fusion for automatic sign language recognition , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[15]  W. Marsden.  I and J , 2012 .

[16]  DaeEun Kim,et al.  Finger-gesture Recognition Glove using Velostat (ICCAS 2011) , 2011, 2011 11th International Conference on Control, Automation and Systems.

[17]  Wu jiangqin,et al.  A simple sign language recognition system based on data glove , 1998, ICSP '98. 1998 Fourth International Conference on Signal Processing (Cat. No.98TH8344).

[18]  Mathias Wilhelm,et al.  Whole hand modeling using 8 wearable sensors: biomechanics for hand pose prediction , 2013, AH.

[19]  Tom G. Zimmerman,et al.  A hand gesture interface device , 1987, CHI '87.

[20]  Paul Lukowicz,et al.  Using multiple sensors for mobile sign language recognition , 2003, Seventh IEEE International Symposium on Wearable Computers, 2003. Proceedings..

[21]  Kunal Kadam,et al.  American Sign Language Interpreter , 2012, 2012 IEEE Fourth International Conference on Technology for Education.

[22]  Surapa Thiemjarus,et al.  An Assistive Body Sensor Network Glove for Speech- and Hearing-Impaired Disabilities , 2011, 2011 International Conference on Body Sensor Networks.