DyFAV: Dynamic Feature Selection and Voting for Real-time Recognition of Fingerspelled Alphabet using Wearables

Recent research has shown that reliable recognition of sign language words and phrases using user-friendly and non-invasive armbands is feasible and desirable. This work provides an analysis and implementation of including fingerspelling recognition (FR) in such systems, which is a much harder problem due to lack of distinctive hand movements. A novel algorithm called DyFAV (Dynamic Feature Selection and Voting) is proposed for this purpose that exploits the fact that fingerspelling has a finite corpus (26 letters for ASL). The system uses an independent multiple agent voting approach to identify letters with high accuracy. The independent voting of the agents ensures that the algorithm is highly parallelizable and thus recognition times can be kept low to suit real-time mobile applications. The results are demonstrated on the entire ASL alphabet corpus for nine people with limited training and average recognition accuracy of 95.36% is achieved which is better than the state-of-art for armband sensors. The mobile, non-invasive, and real time nature of the technology is demonstrated by evaluating performance on various types of Android phones and remote server configurations.

[1]  W. Stokoe,et al.  Sign language structure: an outline of the visual communication systems of the American deaf. 1960. , 1961, Journal of deaf studies and deaf education.

[2]  R. Battison,et al.  Lexical Borrowing in American Sign Language , 1978 .

[3]  Martin L. A. Sternberg.  American Sign Language Dictionary , 1981 .

[4]  Carol Padden,et al.  American Sign Language and the architecture of phonological theory , 1987 .

[5]  Sherman Wilcox,et al.  The phonetics of fingerspelling , 1992 .

[6]  Geoffrey E. Hinton,et al.  Glove-Talk: a neural network interface between a data-glove and a speech synthesizer , 1993, IEEE Trans. Neural Networks.

[7]  Geoffrey E. Hinton,et al.  Glove-TalkII-a neural-network interface which maps gestures to parallel formant speech synthesizer controls , 1997, IEEE Trans. Neural Networks.

[8]  Ming Ouhyoung,et al.  A real-time continuous gesture recognition system for sign language , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[9]  Alex Pentland,et al.  Real-Time American Sign Language Recognition Using Desk and Wearable Computer Based Video , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  Diane Brentari,et al.  FOREIGN VOCABULARY IN SIGN LANGUAGES: A CROSS-LINGUISTIC INVESTIGATION OF WORD FORMATION. Diane Brentari (Ed.). Mahwah, NJ: Erlbaum, 2001. Pp. xx + 186. $49.95 cloth. , 2002, Studies in Second Language Acquisition.

[11]  Annelies Braffort,et al.  Some Issues in Sign Language Processing , 2003, Gesture Workshop.

[12]  C. Padden.  Learning to Fingerspell Twice: Young Signing Children’s Acquisition of Fingerspelling , 2005 .

[13]  Johannes Wagner,et al.  Bi-channel sensor fusion for automatic sign language recognition , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[14]  Ming C. Leu,et al.  American Sign Language word recognition with a sensory glove using artificial neural networks , 2011, Eng. Appl. Artif. Intell..

[15]  Nicolas Pugeault,et al.  Spelling it out: Real-time ASL fingerspelling recognition , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[16]  Mosiuoa M. Sole,et al.  Sign language recognition using the Extreme Learning Machine , 2011, IEEE Africon '11.

[17]  Kongqiao Wang,et al.  A Framework for Hand Gesture Recognition Based on Accelerometer and EMG Sensors , 2011, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[18]  Rafiqul Zaman Khan,et al.  Survey on Gesture Recognition for Hand Image Postures , 2012, Comput. Inf. Sci..

[19]  J. M. Kortelainen,et al.  Multichannel bed pressure sensor for sleep monitoring , 2012, 2012 Computing in Cardiology.

[20]  Azli Yahya,et al.  Sign language recognition system using SEMG and hidden markov model , 2013 .

[21]  Bodo Rosenhahn,et al.  Real-Time Sign Language Recognition Using a Consumer Depth Camera , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[22]  Sethuraman Panchanathan,et al.  Multiresolution Match Kernels for gesture video classification , 2013, 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW).

[23]  Karen Das,et al.  Indian Sign Language Recognition Using Eigen Value Weighted Euclidean Distance Based Classification Technique , 2013, ArXiv.

[24]  Jacir Luiz Bordim,et al.  Accuracy and Efficiency Performance of the ICP Procedure Applied to Sign Language Recognition , 2014, CLEI Electron. J..

[25]  D Karthikeyan,et al.  English Letters Finger Spelling Sign Language Recognition System , 2014 .

[26]  Qiang Li,et al.  A Hand Gesture Recognition Framework and Wearable Gesture-Based Interaction Prototype for Mobile Devices , 2014, IEEE Transactions on Human-Machine Systems.

[27]  Mustapha Hamad,et al.  Sign to letter translator system using a hand glove , 2014, The Third International Conference on e-Technologies and Networks for Development (ICeND2014).

[28]  Koosha Sadeghi,et al.  Enabling Real-Time Collaborative Brain-Mobile Interactive Applications on Volunteer Mobile Devices , 2015, HotWireless@MobiCom.

[29]  Ferat Sahin,et al.  Real-Time American Sign Language Recognition System Using Surface EMG Signal , 2015, 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA).

[30]  Khaled Assaleh,et al.  Glove-Based Continuous Arabic Sign Language Recognition in User-Dependent Mode , 2015, IEEE Transactions on Human-Machine Systems.

[31]  Ayan Banerjee,et al.  MT-Diet: Automated smartphone based diet assessment with infrared images , 2016, 2016 IEEE International Conference on Pervasive Computing and Communications (PerCom).

[32]  Ferat Sahin,et al.  American Sign Language Recognition system by using surface EMG signal , 2016, 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[33]  Ayan Banerjee,et al.  SCEPTRE: A Pervasive, Non-Invasive, and Programmable Gesture Recognition Technology , 2016, IUI.

[34]  João Marcelo X. N. Teixeira,et al.  Evaluating Sign Language Recognition Using the Myo Armband , 2016, 2016 XVIII Symposium on Virtual and Augmented Reality (SVR).