A sliding window approach to natural hand gesture recognition using a custom data glove

This paper explores the recognition of hand gestures based on a data glove equipped with motion, bending and pressure sensors. We selected 31 natural and interaction-oriented hand gestures that can be adopted for general-purpose control of and communication with computing systems. The data glove is custom-built, and contains 13 bend sensors, 7 motion sensors, 5 pressure sensors and a magnetometer. We present the data collection experiment, as well as the design, selection and evaluation of a classification algorithm. As we use a sliding window approach to data processing, our algorithm is suitable for stream data processing. Algorithm selection and feature engineering resulted in a combination of linear discriminant analysis and logistic regression with which we achieve an accuracy of over 98.5% on a continuous data stream scenario. When removing the computationally expensive FFT-based features, we still achieve an accuracy of 98.2%.

[1]  Holger Kenn,et al.  A glove-based gesture interface for wearable computing applications , 2007 .

[2]  Terrence J. Sejnowski,et al.  Parallel Networks that Learn to Pronounce English Text , 1987, Complex Syst..

[3]  Chung-Lin Huang,et al.  Sign language recognition using model-based tracking and a 3D Hopfield neural network , 1998, Machine Vision and Applications.

[4]  S. Kiruthika,et al.  Survey on Hand Gesture Recognition , 2014 .

[5]  L. Benini,et al.  Activity recognition from on-body sensors by classifier fusion: sensor scalability and robustness , 2007, 2007 3rd International Conference on Intelligent Sensors, Sensor Networks and Information.

[6]  Johannes Schöning,et al.  WatchMe: A Novel Input Method Combining a Smartwatch and Bimanual Interaction , 2015, CHI Extended Abstracts.

[7]  Pedro Neto,et al.  Real-time and continuous hand gesture spotting: An approach based on artificial neural networks , 2013, 2013 IEEE International Conference on Robotics and Automation.

[8]  Alois Knoll,et al.  Multimodal Human Activity Recognition for Industrial Manufacturing Processes in Robotic Workcells , 2015, ICMI.

[9]  Manolya Kavakli,et al.  Designing in virtual reality (DesIRe): a gesture-based interface , 2007, DIMEA.

[10]  Piyush Kumar,et al.  Hand Data Glove: A Wearable Real-Time Device for Human- Computer Interaction , 2012 .

[11]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[12]  Parth H. Pathak,et al.  Finger-writing with Smartwatch: A Case for Finger and Hand Gesture Recognition using Smartwatch , 2015, HotMobile.

[13]  Walter Higgins,et al.  A Comparison of Complementary and Kalman Filtering , 1975, IEEE Transactions on Aerospace and Electronic Systems.

[14]  Pavlo Molchanov,et al.  Multi-sensor system for driver's hand-gesture recognition , 2015, 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[15]  Tomoichi Takahashi,et al.  Hand gesture coding based on experiments using a hand gesture interface device , 1991, SGCH.

[16]  Joseph J. LaViola,et al.  Exploring 3D User Interface Technologies for Improving the Gaming Experience , 2015, CHI.

[17]  Vigneshwaran Subbaraju,et al.  The case for smartwatch-based diet monitoring , 2015, 2015 IEEE International Conference on Pervasive Computing and Communication Workshops (PerCom Workshops).

[18]  Christa Sommerer,et al.  Gauntlet: a wearable interface for ubiquitous gaming , 2008, Mobile HCI.

[19]  Jiawei Han,et al.  SRDA: An Efficient Algorithm for Large-Scale Discriminant Analysis , 2008, IEEE Transactions on Knowledge and Data Engineering.

[20]  Susan R. Fussell,et al.  Combining audio and video to predict helpers' focus of attention in multiparty remote collaboration on physical tasks , 2006, ICMI '06.

[21]  Paul Lukowicz,et al.  Wearable Activity Tracking in Car Manufacturing , 2008, IEEE Pervasive Computing.

[22]  Gang Ren,et al.  Freehand gestural text entry for interactive TV , 2013, EuroITV.

[23]  Diane J. Cook,et al.  Activity recognition on streaming sensor data , 2014, Pervasive Mob. Comput..

[24]  Kun-Lung Wu,et al.  General Incremental Sliding-Window Aggregation , 2015, Proc. VLDB Endow..

[25]  Kongqiao Wang,et al.  Hand gesture recognition and virtual game control based on 3D accelerometer and EMG sensors , 2009, IUI.

[26]  Thomas B. Moeslund,et al.  Real-time recognition of hand alphabet gestures using principal component analysis , 1997 .

[27]  Kouichi Murakami,et al.  Gesture recognition using recurrent neural networks , 1991, CHI.

[28]  Yael Edan,et al.  Vision-based hand-gesture applications , 2011, Commun. ACM.

[29]  Wan-Young Chung,et al.  Wristband-Type Driver Vigilance Monitoring System Using Smartwatch , 2015, IEEE Sensors Journal.

[30]  Ralf Salomon,et al.  Gesture recognition for virtual reality applications using data gloves and neural networks , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[31]  Tsung-Han Tsai,et al.  Embedded virtual mouse system by using hand gesture recognition , 2015, 2015 IEEE International Conference on Consumer Electronics - Taiwan.

[32]  N.D. Georganas,et al.  Real-time Vision-based Hand Gesture Recognition Using Haar-like Features , 2007, 2007 IEEE Instrumentation & Measurement Technology Conference IMTC 2007.

[33]  Pekka Siirtola,et al.  Activity recognition using a wrist-worn inertial measurement unit: A case study for industrial assembly lines , 2009, 2009 17th Mediterranean Conference on Control and Automation.

[34]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[35]  Sanjeev Sofat,et al.  Vision Based Hand Gesture Recognition , 2009 .

[36]  Nicolas D. Georganas,et al.  Real-Time Hand Gesture Detection and Recognition Using Bag-of-Features and Support Vector Machine Techniques , 2011, IEEE Transactions on Instrumentation and Measurement.

[37]  Deyou Xu.  A Neural Network Approach for Hand Gesture Recognition in Virtual Reality Driving Training System of SPG , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[38]  Mokhtar M. Hasan,et al.  Hand Gesture Modeling and Recognition using Geometric Features: A Review , 2012 .

[39]  Michal Romaszewski,et al.  Choosing and Modeling the Hand Gesture Database for a Natural User Interface , 2011, Gesture Workshop.

[40]  Parth H. Pathak,et al.  Demo: Finger and Hand Gesture Recognition using Smartwatch , 2015, MobiSys.

[41]  Thomas G. Dietterich.  Machine Learning for Sequential Data: A Review , 2002, SSPR/SPR.

[42]  Bernt Schiele,et al.  A tutorial on human activity recognition using body-worn inertial sensors , 2014, CSUR.

[43]  Ming C. Leu,et al.  American Sign Language word recognition with a sensory glove using artificial neural networks , 2011, Eng. Appl. Artif. Intell..

[44]  Philippe Fuchs,et al.  User-defined gestural interaction: A study on gesture memorization , 2013, 2013 IEEE Symposium on 3D User Interfaces (3DUI).

[45]  Jamie A. Ward.  Activity monitoring : continuous recognition and performance evaluation , 2006 .

[46]  F. Mörchen.  Time series feature extraction for data mining using DWT and DFT , 2003 .

[47]  Piotr Gawron,et al.  Natural hand gestures for human identification in a Human-Computer Interface , 2014, 2014 4th International Conference on Image Processing Theory, Tools and Applications (IPTA).

[48]  Juha Röning,et al.  Behavior modeling in industrial assembly lines using a wrist-worn inertial measurement unit , 2013, J. Ambient Intell. Humaniz. Comput..

[49]  Fabien Moutarde,et al.  Gesture Recognition Using a Depth Camera for Human Robot Collaboration on Assembly Line , 2015 .

[50]  Paul Lukowicz,et al.  Activity Recognition of Assembly Tasks Using Body-Worn Microphones and Accelerometers , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[51]  Paul J. M. Havinga,et al.  Towards detection of bad habits by fusing smartphone and smartwatch sensors , 2015, 2015 IEEE International Conference on Pervasive Computing and Communication Workshops (PerCom Workshops).

[52]  Daniel Borrajo,et al.  A dynamic sliding window approach for activity recognition , 2011, UMAP'11.

[53]  Geehyuk Lee,et al.  Transture: Continuing a Touch Gesture on a Small Screen into the Air , 2015, CHI Extended Abstracts.