SGM-Net: Skeleton-guided multimodal network for action recognition

Abstract Single-modality human action recognition on RGB or skeleton has been extensively studied. Each of these two modalities has its own advantages as well as limitations, because they depict action from different perspectives. The feature of different modalities can complement each other for describing actions. Therefore, it is meaningful to fuse these two modalities using their complementarity for action recognition. However, existing multimodal methods fail to fully exploit the complementarity of RGB and skeleton modalities. In this paper, we propose a Skeleton-Guided Multimodal Network (SGM-Net) for human action recognition. The proposed method takes full use of the complementarity of these two modalities at semantic feature level. From the technical perspective, we introduce a guided block, the key component of SGM-Net. It enables skeleton feature to guide on RGB feature, so that the important RGB information strongly related to the action is enhanced. Moreover, in the guided block, two schemes of correlation operation are explored. We perform a series of ablation experiments to verify the effectiveness of the guided block. The experimental results show that our approach achieves state-of-the-art performance over the existing methods on NTU and Sub-JHMDB datasets.