Real time hand gesture recognition via finger-emphasized multi-scale description

The development of depth cameras, e.g., the Kinect sensor, provides new opportunities for human computer interaction (HCI). Although the Kinect sensor has been extensively applied for human tracking, human action recognition and hand gesture recognition, real time hand gesture recognition is still a challenging problem. In this paper, we propose a new real time hand gesture recognition method. To represent the noisy and articulated hand shape segmented from the Kinect images, a finger emphasized multi-scale descriptor is proposed. To fully utilize hand shape features, this descriptor incorporates three types of parameters of multiple scales, which emphasize the finger features. Hand gesture recognition is then achieved with both DTW algorithm and BP neural network. Extensive experimental results and the comparison with state-of-the-art methods demonstrate that our method is accurate (a 100% accuracy on a challenging hand gesture dataset), efficient (average 0.941ms per frame), and robust to noise, articulations and rigid transformations.

[1]  Pietro Zanuttigh,et al.  Hand gesture recognition with jointly calibrated Leap Motion and depth sensor , 2015, Multimedia Tools and Applications.

[2]  Junsong Yuan,et al.  Robust Part-Based Hand Gesture Recognition Using Kinect Sensor , 2013, IEEE Transactions on Multimedia.

[3]  R. S. Jadon,et al.  A REVIEW OF VISION BASED HAND GESTURES RECOGNITION , 2009 .

[4]  Jianyu Yang,et al.  Invariant multi-scale descriptor for shape representation, matching and retrieval , 2016, Comput. Vis. Image Underst..

[5]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[6]  Radu Horaud,et al.  Point Trajectories and a Smooth Surface Model , 2004, European Conference on Computer Vision.

[7]  Luc Van Gool,et al.  Smart particle filtering for 3D hand tracking , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[9]  Hecht-Nielsen.  Theory of the backpropagation neural network , 1989 .

[10]  Mu-Chun Su,et al.  A fuzzy rule-based approach to spatio-temporal hand gesture recognition , 2000, IEEE Trans. Syst. Man Cybern. Part C.

[11]  Mircea Nicolescu,et al.  Vision-based hand pose estimation: A review , 2007, Comput. Vis. Image Underst..

[12]  Dewen Hu,et al.  Globally Consistent Reconstruction of Ripped-Up Documents , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Pietro Zanuttigh,et al.  Head-mounted gesture controlled interface for human-computer interaction , 2016, Multimedia Tools and Applications.

[14]  Yael Edan,et al.  Vision-based hand-gesture applications , 2011, Commun. ACM.

[15]  Hanqing Lu,et al.  A real-time hand gesture recognition method , 2007, 2011 International Conference on Electronics, Communications and Control (ICECC).

[16]  Qi Ye,et al.  Spatial Attention Deep Net with Partial PSO for Hierarchical Hybrid Hand Pose Estimation , 2016, ECCV.

[17]  Haibin Ling,et al.  Shape Classification Using the Inner-Distance , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Pavlo Molchanov,et al.  Online Detection and Classification of Dynamic Hand Gestures with Recurrent 3D Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Jitendra Malik,et al.  Shape matching and object recognition using shape contexts , 2010, 2010 3rd International Conference on Computer Science and Information Technology.

[20]  Robert Hecht-Nielsen,et al.  Theory of the backpropagation neural network , 1989, International 1989 Joint Conference on Neural Networks.

[21]  Dieter Fox,et al.  Real-time particle filters , 2004, Proceedings of the IEEE.

[22]  Pietro Zanuttigh,et al.  Hand gesture recognition with leap motion and kinect devices , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[23]  Haiying Guan,et al.  Model-based 3D hand posture estimation from a single 2D image , 2002, Image Vis. Comput..

[24]  Narendra Ahuja,et al.  Extraction of 2D Motion Trajectories and Its Application to Hand Gesture Recognition , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  Aaron F. Bobick,et al.  Parametric Hidden Markov Models for Gesture Recognition , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Longin Jan Latecki,et al.  Path Similarity Skeleton Graph Matching , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Eric Foxlin,et al.  Motion Tracking Requirements and Technologies , 2002 .

[28]  Björn Stenger,et al.  Filtering using a tree-based estimator , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[29]  Kaleem Siddiqi,et al.  Hamilton-Jacobi Skeletons , 2002, International Journal of Computer Vision.

[30]  Shaun J. Canavan,et al.  A Multi-Gesture Interaction System Using a 3-D Iris Disk Model for Gaze Estimation and an Active Appearance Model for 3-D Hand Pointing , 2011, IEEE Transactions on Multimedia.