Temporal Context Analysis for Action Recognition in Multi-agent Scenarios

In multi-agent scenarios such as sports videos, multiple actions are played by different players. Such actions do not necessary appear strictly sequentially but can happen in parallel. Approaches which only consider a single stream of actions are not competent to handle such scenarios. The temporal and causal relationships between the action streams such as "concurrence", "mutually exclusion" and "triggering" need to be captured so as to correctly recognize the actions. In this paper, a novel method is presented for action recognition in multi-agent scenarios leveraged by analyzing the relationships among the temporal contextual actions. The multi-streams of actions are modeled by a Dynamic Baysian Network (DBN) containing several temporal processes corresponding to each type of action. Comparing to the Coupled Hidden Markov Model (CHMM), only the necessary interlinks between the temporal processes are built by a structure learning algorithm to capture the salient relationships. Empirical results on real-world video data demonstrate the effectiveness of our proposed method.