Dynamic Gesture Recognition with Pose-based CNN Features derived from videos using LSTM

In this work, we address the problem of dynamic gesture recognition using a pose based video descriptor. The proposed approach takes as input video frames and extracts pose-specific image regions which are further processed by a pre-trained Convolutional Neural Network (CNN) to derive a pose-based descriptor for each frame. A Long Short Term Memory (LSTM) network is trained from scratch for dynamic gesture classification by learning long-term spatiotemporal relations among features. We also demonstrate that only using video data (RGB frames and optical flow) one can design an effective model for recognizing dynamic gestures. We utilize ChaLearn multi-modal gesture challenge dataset [13] and Cambridge hand gesture dataset [18] for evaluation of the proposed algorithm achieving an accuracy of 91.27% and 96% respectively using only RGB data.