A Real-time Multimodal Hand Gesture Recognition via 3D Convolutional Neural Network and Key Frame Extraction

In this paper, we present an approach for hand gesture recognition by 3D Convolutional Neural Network 3D_CNN and key frames extractor algorithm by the fast neural network. Typically, 3D_CNN algorithms classify hand gestures from a number of randomly sampled image sequences. In this work, key frames extracted from static video summarization based clustering method was used as input of 3D_CNN algorithm to improve classification accuracy. Because of expensive computation of video summarization, we propose a fast deep neural network SegNet based video summarization method VSUMM to learn and identify key frames in a video sequence to speed up the computational time to appropriate with a real-time system. We evaluate our proposed algorithms on publish Cambridge gestures dataset and Seven Hand Gestures SHG Dataset. We also experimentally estimate number of key frames per a video sequence in those datasets. This algorithm achieved 94.4% classification accuracy on the Cambridge gestures dataset and 77.71% classification accuracy on the SHG Dataset. The experiment results show that the proposed approach is efficient and outperforms to compare with state-of-art related researches with real-time computational efficiency.