Joint specific and correlated information exploration for multi-view action clustering

Abstract Human action clustering is crucial in many practical applications. However, existing action clustering methods work in a single-view manner, which always ignore the relationships among different views and fail to discover correct clusters as the viewpoint and position change. To address the challenges, we propose a unified framework for multi-view human action clustering. First, we design a new Bag-of-Shared-Words (BoSW) model to discover the view-shared visual words that preserve the consistency among visual words of different views. Then, we obtain a more discriminative feature representation, from which the view correlation can be fully explored. Then, we present a novel JOint INformation boTtleneck (JOINT) algorithm to jointly exploit both the view-specific and view-correlated information to improve the action clustering performance. Specifically, JOINT formulates the problem as minimizing an information loss function, which compresses the actions of each view while jointly preserving the complementary view-specific information and correlated information among views. To solve the proposed objective function, a new sequential procedure is presented to guarantee convergence to a local optimal solution. Extensive experiments on three challenging multi-view single-person and interactive action datasets demonstrate the superiority of our algorithm.