Deep Fisher discriminant learning for mobile hand gesture recognition

Gesture recognition is a challenging problem in the field of biometrics. In this paper, we integrate Fisher criterion into Bidirectional Long-Short Term Memory (BLSTM) network and Bidirectional Gated Recurrent Unit (BGRU),thus leading to two new deep models termed as F-BLSTM and F-BGRU. BothFisher discriminative deep models can effectively classify the gesture based on analyzing the acceleration and angular velocity data of the human gestures. Moreover, we collect a large Mobile Gesture Database (MGD) based on the accelerations and angular velocities containing 5547 sequences of 12 gestures. Extensive experiments are conducted to validate the superior performance of the proposed networks as compared to the state-of-the-art BLSTM and BGRU on MGD database and two benchmark databases (i.e. BUAA mobile gesture and SmartWatch gesture).

[1]  Alois Ferscha,et al.  Gestural interaction in the pervasive computing landscape , 2007, Elektrotech. Informationstechnik.

[2]  Cagatay Catal,et al.  On the use of ensemble of classifiers for accelerometer-based activity recognition , 2015, Appl. Soft Comput..

[3]  Kuldip K. Paliwal,et al.  Bidirectional recurrent neural networks , 1997, IEEE Trans. Signal Process..

[4]  Yoshua Bengio,et al.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling , 2014, ArXiv.

[5]  Eric O. Postma,et al.  Dimensionality Reduction: A Comparative Review , 2008 .

[6]  Chen Chen,et al.  Output Constraint Transfer for Kernelized Correlation Filter in Tracking , 2016, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[7]  Kongqiao Wang,et al.  Hand gesture recognition and virtual game control based on 3D accelerometer and EMG sensors , 2009, IUI.

[8]  Romit Roy Choudhury,et al.  Using mobile phones to write in air , 2011, MobiSys '11.

[9]  Matthew J. Hausknecht,et al.  Beyond short snippets: Deep networks for video classification , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Emiliano Miluzzo,et al.  A survey of mobile phone sensing , 2010, IEEE Communications Magazine.

[11]  Junsong Yuan,et al.  Learning Actionlet Ensemble for 3D Human Action Recognition , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Thuc Dinh Nguyen,et al.  Adaptive Cross-Device Gait Recognition Using a Mobile Accelerometer , 2013, J. Inf. Process. Syst..

[13]  Timo Pylvänäinen,et al.  Accelerometer Based Gesture Recognition Using Continuous HMMs , 2005, IbPRIA.

[14]  Meng Wang,et al.  Image-Based Three-Dimensional Human Pose Recovery by Multiview Locality-Sensitive Sparse Retrieval , 2015, IEEE Transactions on Industrial Electronics.

[15]  Jung-Shyr Wu,et al.  Integrating LCS and SVM for 3D handwriting recognition on handheld devices using accelerometers , 2009, ICC 2009.

[16]  Chen Chen,et al.  Image Reconstruction via Manifold Constrained Convolutional Sparse Coding for Image Sets , 2017, IEEE Journal of Selected Topics in Signal Processing.

[17]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[18]  Günter Hommel,et al.  Velocity Profile Based Recognition of Dynamic Gestures with Discrete Hidden Markov Models , 1997, Gesture Workshop.

[19]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[20]  Daniel Roggen,et al.  Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition , 2016, Sensors.

[21]  Yoshua Bengio,et al.  On the Properties of Neural Machine Translation: Encoder–Decoder Approaches , 2014, SSST@EMNLP.

[22]  Yu Qiao,et al.  A Discriminative Feature Learning Approach for Deep Face Recognition , 2016, ECCV.

[23]  Karandeep Singh,et al.  A Single Accelerometer based Wireless Embedded System for Predefined Dynamic Gesture Recognition , 2009, IHCI.

[24]  Silvio Savarese,et al.  Social LSTM: Human Trajectory Prediction in Crowded Spaces , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Kyungtae Kang,et al.  Intelligent classification of heartbeats for automated real-time ECG monitoring. , 2014, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[26]  Tea Marasovic,et al.  Accelerometer-based gesture classification using principal component analysis , 2011, SoftCOM 2011, 19th International Conference on Software, Telecommunications and Computer Networks.

[27]  Wonyong Sung,et al.  Dynamic hand gesture recognition for wearable devices with low complexity recurrent neural networks , 2016, 2016 IEEE International Symposium on Circuits and Systems (ISCAS).

[28]  Hermann Ney,et al.  LSTM Neural Networks for Language Modeling , 2012, INTERSPEECH.

[29]  S. Eddy.  Hidden Markov models. , 1996, Current opinion in structural biology.

[30]  Greg Mori,et al.  A Hierarchical Deep Temporal Model for Group Activity Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Baochang Zhang,et al.  Adaptive Local Movement Modeling for Robust Object Tracking , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[32]  Guo-Jun Qi,et al.  Differential Recurrent Neural Networks for Action Recognition , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[33]  Yoshua Bengio,et al.  Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding , 2013, INTERSPEECH.

[34]  Paolo Valigi,et al.  Personalizing a smartwatch-based gesture interface with transfer learning , 2014, 2014 22nd European Signal Processing Conference (EUSIPCO).

[35]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[36]  Jun Yu,et al.  Multitask Autoencoder Model for Recovering Human Poses , 2018, IEEE Transactions on Industrial Electronics.

[37]  Yong Du,et al.  Hierarchical recurrent neural network for skeleton based action recognition , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Shyamnath Gollakota,et al.  Contactless Sleep Apnea Detection on Smartphones , 2015, GetMobile Mob. Comput. Commun..

[39]  Nasser Kehtarnavaz,et al.  Fusion of Inertial and Depth Sensor Data for Robust Hand Gesture Recognition , 2014, IEEE Sensors Journal.

[40]  Eliyahu Kiperwasser,et al.  Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations , 2016, TACL.

[41]  Meng Yang,et al.  Large-Margin Softmax Loss for Convolutional Neural Networks , 2016, ICML.

[42]  Shahrokh Valaee,et al.  Accelerometer-based gesture recognition via dynamic-time warping, affinity propagation, & compressive sensing , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[43]  Wei Gao,et al.  Accelerometer-based hand gesture recognition using feature weighted naïve bayesian classifiers and dynamic time warping , 2013, IUI '13 Companion.

[44]  Jing Yang,et al.  Beatbox music phone: gesture-based interactive mobile phone using a tri-axis accelerometer , 2005, 2005 IEEE International Conference on Industrial Technology.

[45]  Jun Rekimoto,et al.  GestureWrist and GesturePad: unobtrusive wearable interaction devices , 2001, Proceedings Fifth International Symposium on Wearable Computers.

[46]  Billur Barshan,et al.  Comparative study on classifying human activities with miniature inertial and magnetic sensors , 2010, Pattern Recognit..

[47]  Yoshua Bengio,et al.  Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , 2015, ICML.

[48]  Paul Lukowicz,et al.  Gesture spotting with body-worn inertial sensors to detect user activities , 2008, Pattern Recognit..

[49]  Meng Wang,et al.  Multimodal Deep Autoencoder for Human Pose Recovery , 2015, IEEE Transactions on Image Processing.

[50]  Zhen Wang,et al.  uWave: Accelerometer-based Personalized Gesture Recognition and Its Applications , 2009, PerCom.

[51]  Zhenyu He.  Accelerometer Based Gesture Recognition Using Fusion Features and SVM , 2011, J. Softw..

[52]  Samy Bengio,et al.  Show and tell: A neural image caption generator , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[53]  Ao Tang,et al.  A Real-Time Hand Posture Recognition System Using Deep Neural Networks , 2015, ACM Trans. Intell. Syst. Technol..

[54]  He Wang,et al.  I am a smartphone and i can tell my user's walking direction , 2014, MobiSys.

[55]  I. J. Jang,et al.  Signal processing of the accelerometer for gesture awareness on handheld devices , 2003, The 12th IEEE International Workshop on Robot and Human Interactive Communication, 2003. Proceedings. ROMAN 2003..

[56]  Ling Shao,et al.  Action Recognition Using 3D Histograms of Texture and A Multi-Class Boosting Classifier , 2017, IEEE Transactions on Image Processing.

[57]  Gang Wang,et al.  Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition , 2016, ECCV.

[58]  Lukás Burget,et al.  Extensions of recurrent neural network language model , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[59]  Daqing Zhang,et al.  Gesture Recognition with a 3-D Accelerometer , 2009, UIC.

[60]  Tapio Seppänen,et al.  Hand gesture recognition of a mobile device user , 2000, 2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No.00TH8532).

[61]  Greg Mori,et al.  Structure Inference Machines: Recurrent Neural Networks for Analyzing Relations in Group Activity Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[62]  Jani Mäntyjärvi,et al.  Accelerometer-based gesture control for a design environment , 2006, Personal and Ubiquitous Computing.

[63]  Baochang Zhang,et al.  Gesture Recognition Benchmark Based on Mobile Phone , 2016, CCBR.

[64]  Bernt Schiele,et al.  A tutorial on human activity recognition using body-worn inertial sensors , 2014, CSUR.

[65]  Chalavadi Krishna Mohan,et al.  Human action recognition in RGB-D videos using motion sequence information and deep learning , 2017, Pattern Recognit..

[66]  Jani Mäntyjärvi,et al.  Online gesture recognition system for mobile interaction , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[67]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[68]  Luiz Eduardo Soares de Oliveira,et al.  Learning features for offline handwritten signature verification using deep convolutional neural networks , 2017, Pattern Recognit..

[69]  Angelo Cangelosi,et al.  Head pose estimation in the wild using Convolutional Neural Networks and adaptive gradient methods , 2017, Pattern Recognit..

[70]  M. Gams,et al.  Comparing Deep and Classical Machine Learning Methods for Human Activity Recognition using Wrist Accelerometer , 2016 .

[71]  Xiaohui Xie,et al.  Co-Occurrence Feature Learning for Skeleton Based Action Recognition Using Regularized Deep LSTM Networks , 2016, AAAI.

[72]  Yasushi Makihara,et al.  Similar gait action recognition using an inertial sensor , 2015, Pattern Recognit..