Dynamic Hand Gesture Recognition from Multi-modal Streams Using Deep Neural Network

Hand gesture is an efficient mean of human computer interaction. However, hand gesture recognition faces many challenges such as low hand resolution, phase variation and viewpoint. As a result, deployment of hand gesture in a practical application of human machine interaction is still very limited. This work aims to increase performance of hand gestures recognition by using multi-modal streams. We propose a method that combines depth, RGB and optical flow in a unified recognition framework. Each stream will go first into a feature extraction component, which is a deep learning model. We then investigate different fusion techniques to combine features from multi-modal streams for final classification. The proposed method is validated on a dataset of twelve gestures collected by ourselves from five different viewpoints. Experimental results show that accuracy of the proposed method using multi-modal streams outperforms ones that use a single stream, particularly for difficult viewpoints.

[1]  Junsong Yuan,et al.  Depth camera based hand gesture recognition and its applications in Human-Computer-Interaction , 2011, 2011 8th International Conference on Information, Communications & Signal Processing.

[2]  Juan Song,et al.  Multimodal Gesture Recognition Using 3-D Convolution and Convolutional LSTM , 2017, IEEE Access.

[3]  Thanh-Hai Tran,et al.  Improving Human Action Recognition with Two-Stream 3D Convolutional Neural Network , 2018, 2018 1st International Conference on Multimedia Analysis and Pattern Recognition (MAPR).

[4]  Zheru Chi,et al.  Realistic Human Action Recognition With Multimodal Feature Selection and Fusion , 2013, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[5]  Junsong Yuan,et al.  Robust hand gesture recognition based on finger-earth mover's distance with a commodity depth camera , 2011, ACM Multimedia.

[6]  Ahmet Gunduz,et al.  Real-time Hand Gesture Detection and Classification Using Convolutional Neural Networks , 2019, 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019).

[7]  Wang Xi,et al.  Deep Learning for Hand Gesture Recognition on Skeletal Data , 2018, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[8]  Anupam Agrawal,et al.  Vision based hand gesture recognition for human computer interaction: a survey , 2012, Artificial Intelligence Review.

[9]  Minrui Fei,et al.  Dynamic Hand Gesture Recognition Based on the Three-Dimensional Spatial Trajectory Feature and Hidden Markov Model , 2018 .

[10]  Thanh-Hai Tran,et al.  Phase synchronization in a manifold space for recognizing dynamic hand gestures from periodic image sequence , 2016, 2016 IEEE RIVF International Conference on Computing & Communication Technologies, Research, Innovation, and Vision for the Future (RIVF).

[11]  Soo-Hyung Kim,et al.  A Real-time Multimodal Hand Gesture Recognition via 3D Convolutional Neural Network and Key Frame Extraction , 2018, ICML 2018.

[12]  Yingli Tian,et al.  Edge Enhanced Depth Motion Map for Dynamic Hand Gesture Recognition , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[13]  Lorenzo Torresani,et al.  Learning Spatiotemporal Features with 3D Convolutional Networks , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[14]  Thi-Lan Le,et al.  Robustness Analysis of 3D Convolutional Neural Network for Human Hand Gesture Recognition , 2019, International Journal of Machine Learning and Computing.

[15]  Andrew Zisserman,et al.  Two-Stream Convolutional Networks for Action Recognition in Videos , 2014, NIPS.

[16]  Alessandro Sperduti,et al.  A Kernel Method for the Optimization of the Margin Distribution , 2008, ICANN.

[17]  Bruce A. Draper,et al.  Gesture Recognition: Focus on the Hands , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[18]  Z. Liu,et al.  A real time system for dynamic hand gesture recognition with a depth sensor , 2012, 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO).

[19]  Mohan M. Trivedi,et al.  Hand Gesture Recognition in Real Time for Automotive Interfaces: A Multimodal Vision-Based Approach and Evaluations , 2014, IEEE Transactions on Intelligent Transportation Systems.

[20]  Thanh-Hai Tran,et al.  A combination of user-guide scheme and kernel descriptor on RGB-D data for robust and realtime hand posture recognition , 2016, Eng. Appl. Artif. Intell..

[21]  Ruzena Bajcsy,et al.  Berkeley MHAD: A comprehensive Multimodal Human Action Database , 2013, 2013 IEEE Workshop on Applications of Computer Vision (WACV).