Action recognition by Latent Duration Model

Abstract Temporal structure has attracted lots of research interests due to its ability to model complex pattern for effective action recognition. Most existing methods exploit temporal structure only in a fixed scale or implicit multiple scales. Although recently some methods attempt towards exploiting the temporal structure and relationship by using the durations of action primitives, they cannot effectively conduct the action recognition and discriminative segments discovery simultaneously. In this paper, we propose a novel action recognition method, named Latent Duration Model (LDM), which is a temporal variant of Deformable Part Model (DPM) with explicit durations and temporal ordering constraints. Three types of latent variables are introduced into LDM. Latent duration variables are used to accommodate intra-class temporal scale variation. Latent location variables and latent representation variables are utilized to help search the most discriminative segments in the durations. For temporal structure and relationship, our model takes into account both temporal order and duration changes between consecutive parts, which are robust and flexible to the variety in motion speeds and view angel changes of action videos. Thus, not only discriminative parts with adaptive durations but also robust pairwise relationship is automatically discovered by our model. The experimental results on Olympic Sports, Hollywood2, UCF50 and HMDB51 datasets show the effectiveness of our proposed model.