A Framework for Hand Gesture Recognition Based on Accelerometer and EMG Sensors

This paper presents a framework for hand gesture recognition based on the information fusion of a three-axis accelerometer (ACC) and multichannel electromyography (EMG) sensors. In our framework, the start and end points of meaningful gesture segments are detected automatically by the intensity of the EMG signals. A decision tree and multistream hidden Markov models are utilized as decision-level fusion to get the final results. For sign language recognition (SLR), experimental results on the classification of 72 Chinese Sign Language (CSL) words demonstrate the complementary functionality of the ACC and EMG sensors and the effectiveness of our framework. Additionally, the recognition of 40 CSL sentences is implemented to evaluate our framework for continuous SLR. For gesture-based control, a real-time interactive system is built as a virtual Rubik's cube game using 18 kinds of hand gestures as control commands. While ten subjects play the game, the performance is also examined in user-specific and user-independent classification. Our proposed framework facilitates intelligent and natural control in gesture-based interaction.

[1]  Shigeo Abe DrEng.  Pattern Classification , 2001, Springer London.

[2]  Kevin P. Murphy,et al.  A coupled HMM for audio-visual speech recognition , 2002, 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[3]  L.J. Hadjileontiadis,et al.  Evaluation of surface EMG features for the recognition of American Sign Language gestures , 2006, 2006 International Conference of the IEEE Engineering in Medicine and Biology Society.

[4]  Alex Pentland,et al.  Real-Time American Sign Language Recognition Using Desk and Wearable Computer Based Video , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Tamer Shanableh,et al.  Spatio-Temporal Feature-Extraction Techniques for Isolated Gesture Recognition in Arabic Sign Language , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[6]  Timo Pylvänäinen,et al.  Accelerometer Based Gesture Recognition Using Continuous HMMs , 2005, IbPRIA.

[7]  Li Qiang,et al.  Test-Retest Repeatability of Surface Electromyography Measurement for Hand Gesture , 2008, 2008 2nd International Conference on Bioinformatics and Biomedical Engineering.

[8]  Biing-Hwang Juang,et al.  Fundamentals of speech recognition , 1993, Prentice Hall signal processing series.

[9]  Jani Mäntyjärvi,et al.  Accelerometer-based gesture control for a design environment , 2006, Personal and Ubiquitous Computing.

[10]  Adrian D. C. Chan,et al.  A Gaussian mixture model based classification scheme for myoelectric control of powered upper limb prostheses , 2005, IEEE Transactions on Biomedical Engineering.

[11]  Jean-Philippe Thiran,et al.  Dynamic modality weighting for multi-stream hmms inaudio-visual speech recognition , 2008, ICMI '08.

[12]  David G. Stork,et al.  Pattern Classification , 1973 .

[13]  Kongqiao Wang,et al.  Hand Gesture Recognition Research Based on Surface EMG Sensors and 2D-accelerometers , 2007, 2007 11th IEEE International Symposium on Wearable Computers.

[14]  Elisabeth André,et al.  EMG-based hand gesture recognition for realtime biosignal interfacing , 2008, IUI '08.

[15]  Weihua Sheng,et al.  Wearable Sensor-Based Hand Gesture and Daily Activity Recognition for Robot-Assisted Living , 2011, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[16]  Huosheng Hu,et al.  Myoelectric control systems - A survey , 2007, Biomed. Signal Process. Control..

[17]  Vasiliki Kosmidou,et al.  Sign Language Recognition Using Intrinsic-Mode Sample Entropy on sEMG and Accelerometer Data , 2009, IEEE Transactions on Biomedical Engineering.

[18]  D.M. Sherrill,et al.  A neural network approach to monitor motor activities , 2002, Proceedings of the Second Joint 24th Annual Conference and the Annual Fall Meeting of the Biomedical Engineering Society] [Engineering in Medicine and Biology.

[19]  Johannes Wagner,et al.  Bi-channel sensor fusion for automatic sign language recognition , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[20]  Desney S. Tan,et al.  Demonstrating the feasibility of using forearm electromyography for muscle-computer interfaces , 2008, CHI.

[21]  H. Manabe,et al.  Multi-stream HMM for EMG-based speech recognition , 2004, The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[22]  Paul Lukowicz,et al.  Using multiple sensors for mobile sign language recognition , 2003, Seventh IEEE International Symposium on Wearable Computers, 2003. Proceedings..

[23]  Abdelmajid Ben Hamadou,et al.  Off-line handwritten word recognition using multi-stream hidden Markov models , 2010, Pattern Recognit. Lett..

[24]  Khaled Assaleh,et al.  Vision-based system for continuous Arabic Sign Language recognition in user dependent mode , 2008, 2008 5th International Symposium on Mechatronics and Its Applications.

[25]  Daniel Kelly,et al.  A framework for continuous multimodal sign language recognition , 2009, ICMI-MLMI '09.

[26]  Xiao Hu,et al.  Multivariate AR modeling of electromyography for the classification of upper arm movements , 2004, Clinical Neurophysiology.

[27]  Jani Mäntyjärvi,et al.  Enabling fast and effortless customisation in accelerometer based gesture interaction , 2004, MUM '04.

[28]  Sadaoki Furui,et al.  A stream-weight optimization method for multi-stream HMMs based on likelihood value normalization , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[29]  A. Al-Jumaily,et al.  Channel and Feature Selection in Multifunction Myoelectric Control , 2007, 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[30]  Enrico Costanza,et al.  Toward subtle intimate interfaces for mobile devices using an EMG controller , 2005, CHI.

[31]  Dimitris N. Metaxas,et al.  ASL recognition based on a coupling between HMMs and 3D motion analysis , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[32]  Kevin B. Englehart,et al.  A wavelet-based continuous classification scheme for multifunction myoelectric control , 2001, IEEE Transactions on Biomedical Engineering.

[33]  Scott Axelrod,et al.  Maximum entropy and MCE based HMM stream weight estimation for audio-visual ASR , 2002, 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[34]  Wen Gao,et al.  Large vocabulary sign language recognition based on fuzzy decision trees , 2004, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[35]  Chengjun Liu,et al.  Robust coding schemes for indexing and retrieval from large face databases , 2000, IEEE Trans. Image Process..

[36]  Khaled Assaleh,et al.  Vision-based system for continuous Arabic Sign Language recognition in user dependent mode , 2008, 2008 5th International Symposium on Mechatronics and Its Applications.

[37]  K.R. Wheeler,et al.  Gesture-based control and EMG decomposition , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).