Gesture Recognition With the Linear Optical Sensor and Recurrent Neural Networks

In this paper, the optical linear sensor, a representative of low-resolution sensors, was investigated in the multiclass recognition of near-field hand gestures. The recurrent neural network (RNN) with a gated recurrent unit (GRU) memory cell was utilized as a gestures classifier. A set of 27 gestures was collected from a group of volunteers. The 27 000 sequences obtained were divided into training, validation, and test subsets. The primary research goal was to define the most appropriate model architecture in terms of the accurate recognition of each gesture. An additional aim of the research was to investigate the kind of input data, i.e., raw data or preprocessed (feature) data, which generally produces better results. Therefore, three datasets were generated: raw data, simple features data, and high-level features data. (This includes information about hand poses which are already recognized.) The random search method was applied to achieve hyperparameter optimization to find the best possible topology for the neural network. The analysis performed shows that selected models were characterized by a test score at a level of 96.89% for the raw data, 95.75% for simple features, and 93.38% for high-level features. Results indicate that the direct use of raw data obtained from the optical linear sensor evaluated on the RNN with GRU memory cells allows for the reliable recognition of even complex gestures. Therefore, such solutions may have the potential to serve as a support to or as an alternative to video-based sensors especially for mobile devices.

[1]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[2]  Jin Woo Kim,et al.  A Hand Gesture Recognition Sensor Using Reflected Impulses , 2017, IEEE Sensors Journal.

[3]  Gerhard Tröster,et al.  Real time gesture recognition using continuous time recurrent neural networks , 2007, BODYNETS.

[4]  Sheng Kai Tang,et al.  Virtual Mouse: A Low Cost Proximity-Based Gestural Pointing Device , 2011, HCI.

[5]  Jonas Beskow,et al.  Gesture Recognition System for Isolated Sign Language Signs , 2016 .

[6]  Xilin Chen,et al.  Two streams Recurrent Neural Networks for Large-Scale Continuous Gesture Recognition , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[7]  Sébastien Marcel,et al.  Hand gesture recognition using input-output hidden Markov models , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[8]  Surendra Ranganath,et al.  Gesture recognition via pose classification , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[9]  Kouichi Murakami,et al.  Gesture recognition using recurrent neural networks , 1991, CHI.

[10]  Sung Wonyong,et al.  Dynamic hand gesture recognition for wearable devices with low complexity recurrent neural networks , 2016 .

[11]  Krzysztof Czuszynski,et al.  Pose classification in the gesture recognition using the linear optical sensor , 2017, 2017 10th International Conference on Human System Interactions (HSI).

[12]  Jorge Nocedal,et al.  On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima , 2016, ICLR.

[13]  Seong Jin Yun,et al.  Low-power motion gesture sensor with a partially open cavity package. , 2016, Optics express.

[14]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[15]  Xinyu Wu,et al.  Rapid recognition of dynamic hand gestures using leap motion , 2015, 2015 IEEE International Conference on Information and Automation.

[16]  José García Rodríguez,et al.  A recurrent neural network based Schaeffer gesture recognition system , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[17]  Tao Shen,et al.  DiSAN: Directional Self-Attention Network for RNN/CNN-free Language Understanding , 2017, AAAI.

[18]  Jakub Galka,et al.  Inertial Motion Sensing Glove for Sign Language Gesture Acquisition and Recognition , 2016, IEEE Sensors Journal.

[19]  Stefan Wermter,et al.  Gesture Recognition with a Convolutional Long Short-Term Memory Recurrent Neural Network , 2016, ESANN.

[20]  Houqiang Li,et al.  Sign language recognition with long short-term memory , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[21]  M. Maraqa,et al.  Recognition of Arabic Sign Language (ArSL) using recurrent neural networks , 2008, 2008 First International Conference on the Applications of Digital Information and Web Technologies (ICADIWT).

[22]  Pichao Wang,et al.  Action Recognition Based on Joint Trajectory Maps with Convolutional Neural Networks , 2018, Knowl. Based Syst..

[23]  Yoshua Bengio,et al.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling , 2014, ArXiv.

[24]  Thad Starner,et al.  Visual Recognition of American Sign Language Using Hidden Markov Models. , 1995 .

[25]  Chris Eliasmith,et al.  Hyperopt: a Python library for model selection and hyperparameter optimization , 2015 .

[26]  Giulio Paci,et al.  A Multi-scale Approach to Gesture Detection and Recognition , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[27]  Stefan Wermter,et al.  An analysis of Convolutional Long Short-Term Memory Recurrent Neural Networks for gesture recognition , 2017, Neurocomputing.

[28]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[29]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[30]  Hiroomi Hikawa,et al.  Real time gesture recognition system using posture classifier and Jordan recurrent neural network , 2011, The 2012 International Joint Conference on Neural Networks (IJCNN).

[31]  Suranga Nanayakkara,et al.  zSense: Enabling Shallow Depth Gesture Recognition for Greater Input Expressivity on Smart Wearables , 2015, CHI.

[32]  Razvan Pascanu,et al.  On the difficulty of training recurrent neural networks , 2012, ICML.

[33]  Brandon M. Greenwell,et al.  Hands-On Machine Learning with R , 2019 .

[34]  Thomas S. Huang,et al.  Constructing finite state machines for fast gesture recognition , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[35]  Hui Wang,et al.  Multilingual Convolutional, Long Short-Term Memory, Deep Neural Networks for Low Resource Speech Recognition , 2017 .

[36]  Peter Vamplew,et al.  Recognition and anticipation of hand motions using a recurrent neural network , 1995, Proceedings of ICNN'95 - International Conference on Neural Networks.

[37]  Renqiang Xie,et al.  Accelerometer-Based Hand Gesture Recognition by Neural Network and Similarity Matching , 2016, IEEE Sensors Journal.

[38]  Sebastian Otte,et al.  Local Feature Based Online Mode Detection with Recurrent Neural Networks , 2012, 2012 International Conference on Frontiers in Handwriting Recognition.

[39]  Meredith Ringel Morris,et al.  User-defined gestures for surface computing , 2009, CHI.

[40]  Sepp Hochreiter,et al.  Self-Normalizing Neural Networks , 2017, NIPS.

[41]  Krzysztof Czuszynski,et al.  The passive operating mode of the linear optical gesture sensor , 2017, ArXiv.

[42]  Pichao Wang,et al.  Action Recognition Based on Joint Trajectory Maps Using Convolutional Neural Networks , 2016, ACM Multimedia.

[43]  Krzysztof Czuszynski,et al.  Analysis of Properties of an Active Linear Gesture Sensor , 2017 .

[44]  Ching-Hua Chuan,et al.  American Sign Language Recognition Using Leap Motion Sensor , 2014, 2014 13th International Conference on Machine Learning and Applications.

[45]  Peter James Vial,et al.  Centroid tracking based dynamic hand gesture recognition using discrete Hidden Markov Models , 2017, Neurocomputing.