Dual-codebook learning and hierarchical transfer for cross-view action recognition

Abstract. We focus on the challenging cross-view action recognition problem. The key to this problem is to find the correspondence between two different views, which is realized in two stages. First, we construct a dual-codebook for the two views, which contains one codebook for each view. Each codeword in one codebook has a corresponding codeword in the other codebook, whereas traditional methods implement independent codebooks for the views. We propose an effective coclustering algorithm based on seminonnegative matrix factorization to derive the dual-codebook. Additionally, to represent actions in one view, unlike most other methods using the codebook of that view only, we also exploit the codebook-specific information from the other view. Thus, we construct mapped-codebooks via codebook transformation, complementing the codebook-to-codebook correspondence within the dual-codebook. In the second stage, observing that the temporal relationship between action segments within an action is view invariant, we further propose a hierarchical transfer framework based on a temporal structure that can effectively capture such action-segment temporal relationship at multiple timescales, which is more discriminative than the usual video-level transfer strategy. Extensive experimental results on the INRIA xmas motion acquisition sequences and West Virginia University datasets demonstrate superiority of the proposed method compared with state-of-the-art approaches.