Robust methods for hand gesture spotting and recognition using Hidden Markov Models and Conditional Random Fields

This paper proposes an automatic method that handles hand gesture spotting and recognition simultaneously. To spot meaningful gestures of numbers (0–9) accurately, a stochastic method for designing a non-gesture model with Hidden Markov Models (HMMs) versus Conditional Random Fields (CRFs) is proposed without training data. The non-gesture model provides a confidence measure that is used as an adaptive threshold to find the start and the end point of meaningful gestures, which are embedded in the input video stream. To reduce the states number of the non-gesture model with HMMs, similar probability distributions states are merged based on relative entropy measure. Additionally, the weights of self-transition feature functions are increased for short gesture to further improve the accuracy of gesture spotting and recognition with CRFs. Experimental results show that; the proposed method can successfully spot and recognize meaningful gestures with 93.31% and 90.49% reliability for HMMs and CRFs respectively. In addition, the model inference by HMMs are faster and the saving time is 66.42% using relative entropy. The reliability of CRFs method is improved from 86.12% to 90.49% using short gesture detector.

[1]  Narendra Ahuja,et al.  Robust video shot change detection , 1998, 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No.98EX175).

[2]  Alexander H. Waibel,et al.  Natural human-robot interaction using speech, head pose and gestures , 2004, 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566).

[3]  Ayoub Al-Hamadi,et al.  Hand Gesture Spotting Based on 3D Dynamic Features Using Hidden Markov Models , 2009, FGIT-SIP.

[4]  Jin-Hyung Kim,et al.  An HMM-Based Threshold Model Approach for Gesture Recognition , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Ayoub Al-Hamadi,et al.  Hand Gesture Recognition Based on Combined Features Extraction , 2009 .

[6]  Ayoub Al-Hamadi,et al.  Real-Time Capable System for Hand Gesture Recognition Using Hidden Markov Models in Stereo Color Image Sequences , 2008, J. WSCG.

[7]  Ayoub Al-Hamadi,et al.  Posture and Gesture Recognition for Human-Computer Interaction , 2009 .

[8]  Ayoub Al-Hamadi,et al.  Hand trajectory-based gesture spotting and recognition using HMM , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[9]  Alex Pentland,et al.  sing Interpolated Views , 1996 .

[10]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[11]  Ayoub Al-Hamadi,et al.  Discriminative Models-Based Hand Gesture Recognition , 2009, 2009 Second International Conference on Machine Vision.

[12]  Ayoub Al-Hamadi,et al.  A Novel System for Automatic Hand Gesture Spotting and Recognition in Stereo Color Image , 2009, J. WSCG.

[13]  Stan Sclaroff,et al.  A Unified Framework for Gesture Recognition and Spatiotemporal Gesture Segmentation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  A. Al-Hamadi,et al.  Spatio-temporal feature extraction-based hand gesture recognition for isolated American Sign Language and Arabic numbers , 2009, 2009 Proceedings of 6th International Symposium on Image and Signal Processing and Analysis.

[15]  Stan Sclaroff,et al.  Sign Language Spotting with a Threshold Model Based on Conditional Random Fields , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Ayoub Al-Hamadi,et al.  A Robust Method for Hand Gesture Segmentation and Recognition Using Forward Spotting Scheme in Conditional Random Fields , 2010, 2010 20th International Conference on Pattern Recognition.

[17]  Seong-Whan Lee,et al.  Robust Spotting of Key Gestures from Whole Body Motion Sequence , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[18]  R. Oka,et al.  Spotting Recognition of Human Gestures from Motion Images , 1994 .

[19]  A. Al-Hamadi,et al.  Gesture Recognition for Alphabets from Hand Motion Trajectory Using Hidden Markov Models , 2007, 2007 IEEE International Symposium on Signal Processing and Information Technology.

[20]  Ayoub Al-Hamadi,et al.  Improving Hand Gesture Recognition Using 3D Combined Features , 2009, 2009 Second International Conference on Machine Vision.

[21]  Deyou Xu.  A Neural Network Approach for Hand Gesture Recognition in Virtual Reality Driving Training System of SPG , 2006, 18th International Conference on Pattern Recognition (ICPR'06).