Trajectory image based dynamic gesture recognition with convolutional neural networks

Robust dynamic gesture recognition algorithm is of great value for kinds of intelligent interactive systems. Most current researches on this field are based on trajectory time-series, which is unstable and limited. In this paper, we proposed a novel method to realize dynamic gesture recognition by analyzing the static trajectory images with Convolutional Neural Networks (CNN). First of all, a new motion-capture device named Leap Motion is used to track fingertip positions. An effective gesture spotting algorithm is applied to identify the start/end points of dynamic gestures. Then, we map the 3D fingertip coordinates to an image acquisition window frame by frame to get the corresponding trajectory images. After a series of preprocessing steps, the normalized trajectory images are fed to a CNN model. We test the performance of the proposed method on a self-built database, and experimental results show the effectiveness for dynamic gestures recognition of numbers 0-9, with the average recognition rate up to 98.8%.

[1]  Yoshua Bengio,et al.  Convolutional networks for images, speech, and time series , 1998 .

[2]  Z. Liu,et al.  A real time system for dynamic hand gesture recognition with a depth sensor , 2012, 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO).

[3]  Hong Li,et al.  Model-based segmentation and recognition of dynamic gestures in continuous video streams , 2011, Pattern Recognit..

[4]  Pietro Zanuttigh,et al.  Hand gesture recognition with leap motion and kinect devices , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[5]  Dushyant Kumar Singh.  Recognizing hand gestures for human computer interaction , 2015, 2015 International Conference on Communications and Signal Processing (ICCSP).

[6]  H. Seyedarabi,et al.  Real-time dynamic hand gesture recognition using hidden Markov models , 2013, 2013 8th Iranian Conference on Machine Vision and Image Processing (MVIP).

[7]  Yi Li,et al.  Dynamic hand gesture recognition using hidden Markov models , 2012, 2012 7th International Conference on Computer Science & Education (ICCSE).

[8]  Luc Van Gool,et al.  Combining RGB and ToF cameras for real-time 3D hand gesture interaction , 2011, WACV.

[9]  Sergio Escalera,et al.  Featureweighting in dynamic timewarping for gesture recognition in depth data , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[10]  Hui Li,et al.  Kinect Based Dynamic Hand Gesture Recognition Algorithm Research , 2012, 2012 4th International Conference on Intelligent Human-Machine Systems and Cybernetics.

[11]  Sukhan Lee,et al.  Kinect based calling gesture recognition for taking order service of elderly care robot , 2014, The 23rd IEEE International Symposium on Robot and Human Interactive Communication.

[12]  Narendra Ahuja,et al.  Recognizing hand gesture using motion trajectories , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[13]  Deyou Xu.  A Neural Network Approach for Hand Gesture Recognition in Virtual Reality Driving Training System of SPG , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[14]  Pavlo Molchanov,et al.  Hand gesture recognition with 3D convolutional neural networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[15]  Xinyu Wu,et al.  Dynamic gesture recognition using 3D trajectory , 2014, 2014 4th IEEE International Conference on Information Science and Technology.

[16]  Ayoub Al-Hamadi,et al.  Hand trajectory-based gesture spotting and recognition using HMM , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[17]  Ferdinando Silvestro Samaria,et al.  Face recognition using Hidden Markov Models , 1995 .

[18]  Ayoub Al-Hamadi,et al.  Real-Time Capable System for Hand Gesture Recognition Using Hidden Markov Models in Stereo Color Image Sequences , 2008, J. WSCG.

[19]  Manuela M. Veloso,et al.  Depth camera based indoor mobile robot localization and navigation , 2012, 2012 IEEE International Conference on Robotics and Automation.

[20]  Zhiquan Feng,et al.  Dynamic Gesture Recognition Based on Fusing Frame Images , 2013, 2013 Fourth International Conference on Intelligent Systems Design and Engineering Applications.

[21]  Parul Vashist.  Dynamic Hand Gesture Recognition Using Neural Networks , 2013 .

[22]  Ho-Joon Kim,et al.  Dynamic hand gesture recognition using a CNN model with 3D receptive fields , 2008, 2008 International Conference on Neural Networks and Signal Processing.