A Hand Gesture Recognition Framework and Wearable Gesture-Based Interaction Prototype for Mobile Devices

An algorithmic framework is proposed to process acceleration and surface electromyographic (SEMG) signals for gesture recognition. It includes a novel segmentation scheme, a score-based sensor fusion scheme, and two new features. A Bayes linear classifier and an improved dynamic time-warping algorithm are utilized in the framework. In addition, a prototype system, including a wearable gesture sensing device (embedded with a three-axis accelerometer and four SEMG sensors) and an application program with the proposed algorithmic framework for a mobile phone, is developed to realize gesture-based real-time interaction. With the device worn on the forearm, the user is able to manipulate a mobile phone using 19 predefined gestures or even personalized ones. Results suggest that the developed prototype responded to each gesture instruction within 300 ms on the mobile phone, with the average accuracy of 95.0% in user-dependent testing and 89.6% in user-independent testing. Such performance during the interaction testing, along with positive user experience questionnaire feedback, demonstrates the utility of the framework.

[1]  Øyvind Stavdahl,et al.  A multi-modal approach for hand motion classification using surface EMG and accelerometers , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[2]  Xiang Chen,et al.  A prototype of gesture-based interface , 2011, Mobile HCI.

[3]  Hans-Werner Gellersen,et al.  GesturePIN: using discrete gestures for associating mobile devices , 2010, Mobile HCI.

[4]  Kun Hu,et al.  Analysis of EMG and Acceleration Signals for Quantifying the Effects of Deep Brain Stimulation in Parkinson’s Disease , 2011, IEEE Transactions on Biomedical Engineering.

[5]  Li Wei,et al.  Fast time series classification using numerosity reduction , 2006, ICML.

[6]  Zhen Wang,et al.  uWave: Accelerometer-based Personalized Gesture Recognition and Its Applications , 2009, PerCom.

[7]  Ruize Xu,et al.  MEMS Accelerometer Based Nonspecific-User Hand Gesture Recognition , 2012, IEEE Sensors Journal.

[8]  Kongqiao Wang,et al.  A Sign-Component-Based Framework for Chinese Sign Language Recognition Using Accelerometer and sEMG Data , 2012, IEEE Transactions on Biomedical Engineering.

[9]  Kenji Araki,et al.  A Japanese Input Method for Mobile Terminals Using Surface EMG Signals , 2008, JSAI.

[10]  Weihua Sheng,et al.  Wearable Sensor-Based Hand Gesture and Daily Activity Recognition for Robot-Assisted Living , 2011, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[11]  Kongqiao Wang,et al.  A Framework for Hand Gesture Recognition Based on Accelerometer and EMG Sensors , 2011, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[12]  Desney S. Tan,et al.  Making muscle-computer interfaces more practical , 2010, CHI.

[13]  Desney S. Tan,et al.  Enabling always-available input with muscle-computer interfaces , 2009, UIST '09.

[14]  Desney S. Tan,et al.  Demonstrating the feasibility of using forearm electromyography for muscle-computer interfaces , 2008, CHI.

[15]  Ned Jenkinson,et al.  Rapid tremor frequency assessment with the iPhone accelerometer. , 2011, Parkinsonism & related disorders.

[16]  Shahrokh Valaee,et al.  A Novel Accelerometer-based Gesture Recognition System by , 2010 .

[17]  S. S. Joshi,et al.  Brain–Muscle–Computer Interface: Mobile-Phone Prototype Development and Testing , 2011, IEEE Transactions on Information Technology in Biomedicine.

[18]  Timo Pylvänäinen,et al.  Accelerometer Based Gesture Recognition Using Continuous HMMs , 2005, IbPRIA.

[19]  Yue Chen,et al.  SEMG Analysis Basing on AR Model and Bayes Taxonomy , 2010 .

[20]  Guanglin Li,et al.  Performance of various EMG features in identifying ARM movements for control of multifunctional prostheses , 2009, 2009 IEEE Youth Conference on Information, Computing and Telecommunication.

[21]  Elisabeth André,et al.  EMG-based hand gesture recognition for realtime biosignal interfacing , 2008, IUI '08.

[22]  Jeen-Shing Wang,et al.  An Accelerometer-Based Digital Pen With a Trajectory Recognition Algorithm for Handwritten Digit and Gesture Recognition , 2012, IEEE Transactions on Industrial Electronics.

[23]  A. Phinyomark,et al.  Application of Wavelet Analysis in EMG Feature Extraction for Pattern Classification , 2011 .