A Component-Based Vocabulary-Extensible Sign Language Gesture Recognition Framework

Sign language recognition (SLR) can provide a helpful tool for the communication between the deaf and the external world. This paper proposed a component-based vocabulary extensible SLR framework using data from surface electromyographic (sEMG) sensors, accelerometers (ACC), and gyroscopes (GYRO). In this framework, a sign word was considered to be a combination of five common sign components, including hand shape, axis, orientation, rotation, and trajectory, and sign classification was implemented based on the recognition of five components. Especially, the proposed SLR framework consisted of two major parts. The first part was to obtain the component-based form of sign gestures and establish the code table of target sign gesture set using data from a reference subject. In the second part, which was designed for new users, component classifiers were trained using a training set suggested by the reference subject and the classification of unknown gestures was performed with a code matching method. Five subjects participated in this study and recognition experiments under different size of training sets were implemented on a target gesture set consisting of 110 frequently-used Chinese Sign Language (CSL) sign words. The experimental results demonstrated that the proposed framework can realize large-scale gesture set recognition with a small-scale training set. With the smallest training sets (containing about one-third gestures of the target gesture set) suggested by two reference subjects, (82.6 ± 13.2)% and (79.7 ± 13.4)% average recognition accuracy were obtained for 110 words respectively, and the average recognition accuracy climbed up to (88 ± 13.7)% and (86.3 ± 13.7)% when the training set included 50~60 gestures (about half of the target gesture set). The proposed framework can significantly reduce the user’s training burden in large-scale gesture recognition, which will facilitate the implementation of a practical SLR system.

[1]  Qiang Li,et al.  A Hand Gesture Recognition Framework and Wearable Gesture-Based Interaction Prototype for Mobile Devices , 2014, IEEE Transactions on Human-Machine Systems.

[2]  H. S. Wolff,et al.  iRun: Horizontal and Vertical Shape of a Region-Based Graph Compression , 2022, Sensors.

[3]  Surendra Ranganath,et al.  Automatic Sign Language Analysis: A Survey and the Future beyond Lexical Meaning , 2005, IEEE Trans. Pattern Anal. Mach. Intell..

[4]  Kongqiao Wang,et al.  A Framework for Hand Gesture Recognition Based on Accelerometer and EMG Sensors , 2011, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[5]  A. Phinyomark,et al.  Evaluation of EMG feature extraction for hand movement recognition based on Euclidean distance and standard deviation , 2010, ECTI-CON2010: The 2010 ECTI International Confernce on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology.

[6]  Biing-Hwang Juang,et al.  Fundamentals of speech recognition , 1993, Prentice Hall signal processing series.

[7]  Joseph J. LaViola,et al.  Breaking the status quo: Improving 3D gesture recognition with spatially convenient input devices , 2010, 2010 IEEE Virtual Reality Conference (VR).

[8]  Wen Gao,et al.  A novel approach to automatically extracting basic units from Chinese sign language , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[9]  Diego G. S. Santos,et al.  HAGR-D: A Novel Approach for Gesture Recognition with Depth Maps , 2015, Sensors.

[10]  Wen Gao,et al.  Large vocabulary sign language recognition based on fuzzy decision trees , 2004, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[11]  Changsheng Xu,et al.  Discriminative Exemplar Coding for Sign Language Recognition With Kinect , 2013, IEEE Transactions on Cybernetics.

[12]  Kongqiao Wang,et al.  A Sign-Component-Based Framework for Chinese Sign Language Recognition Using Accelerometer and sEMG Data , 2012, IEEE Transactions on Biomedical Engineering.

[13]  Wen Gao,et al.  An approach based on phonemes to large vocabulary Chinese sign language recognition , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[14]  Shigeo Abe DrEng Pattern Classification , 2001, Springer London.

[15]  Francesco Camastra,et al.  LVQ-Based Hand Gesture Recognition Using a Data Glove , 2012, WIRN.

[16]  Xu Zhang,et al.  Random Forest-Based Recognition of Isolated Sign Language Subwords Using Data from Accelerometers and Surface Electromyographic Sensors , 2016, Sensors.

[17]  Daniel Kelly,et al.  A framework for continuous multimodal sign language recognition , 2009, ICMI-MLMI '09.

[18]  Xiang Chen,et al.  Interpreting sign components from accelerometer and sEMG data for automatic sign language recognition , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[19]  Huosheng Hu,et al.  Myoelectric control systems - A survey , 2007, Biomed. Signal Process. Control..

[20]  Aiping Liu,et al.  A Novel Phonology- and Radical-Coded Chinese Sign Language Recognition Framework Using Accelerometer and Surface Electromyography Sensors , 2015, Sensors.

[21]  Vasiliki Kosmidou,et al.  Sign Language Recognition Using Intrinsic-Mode Sample Entropy on sEMG and Accelerometer Data , 2009, IEEE Transactions on Biomedical Engineering.

[22]  Tinne Tuytelaars,et al.  Towards sign language recognition based on body parts relations , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[23]  Zhen Wang,et al.  uWave: Accelerometer-based Personalized Gesture Recognition and Its Applications , 2009, PerCom.

[24]  Roozbeh Jafari,et al.  Real-time American Sign Language Recognition using wrist-worn motion and surface EMG sensors , 2015, 2015 IEEE 12th International Conference on Wearable and Implantable Body Sensor Networks (BSN).

[25]  Liya Ding,et al.  Modelling and recognition of the linguistic components in American Sign Language , 2009, Image Vis. Comput..

[26]  Xia Sun,et al.  Similarity Matching-Based Extensible Hand Gesture Recognition , 2015, IEEE Sensors Journal.