Action Recognition using Deep Convolutional Neural Networks and Compressed Spatio-Temporal Pose Encodings

Convolutional neural networks have recently shown proficiency atrecognizing actions in RGB video. Existing models are gener-ally very deep, requiring large amounts of data to train effectively.Moreover, they rely mainly on global appearance and could poten-tially underperform in single-environment applications, such as asports event. To overcome these limitations, we propose to short-cut spatial learning by leveraging the activations within a humanpose estimation network. The proposed framework integrates ahuman pose estimation network with a convolutional classifier viacompressed encodings of pose activations. When evaluated onUTD-MHAD, a 27-class multimodal dataset, the pose-based RGBaction recognition model achieves a classification accuracy of 98.4%in a subject-specific experiment and outperforms a baseline methodthat fuses depth and inertial sensor data.