Pair-Activity Analysis From Video Using Qualitative Trajectory Calculus

The automated analysis of interacting objects or people from video has many uses, including the recognition of activities, and the identification of prototypical or unusual behaviors. Existing techniques generally use temporal sequences of quantifiable real-valued features, such as object position or orientation; however, more recently, qualitative representations have been proposed. In this paper, we present a novel and robust qualitative method, which can be used for both the classification and the clustering of pair-activities. We use qualitative trajectory calculus (<inline-formula> <tex-math notation="LaTeX">$QTC$ </tex-math></inline-formula>) to represent the relative motion between two objects and encode their interactions as a trajectory of <inline-formula> <tex-math notation="LaTeX">$QTC$ </tex-math></inline-formula> states. A key element is a general and robust means of determining the sequence similarity, which we term <italic>Normalized Weighted Sequence Alignment</italic>; we show that this is an effective metric for both recognition and clustering problems. We have evaluated our method across three different data sets, and have shown that it outperforms the state-of-the-art quantitative methods, achieving an error rate of no more than 4.1% for recognition, and cluster purities higher than 90%. Our motivation originates from an interest in automated analysis of animal behaviors, and we present a comprehensive video data set of fish behaviors (<italic>Gasterosteus aculeatus</italic>), collected from lab-based experiments.