Categorizing bi-object video activities using bag of segments and causality features

We address the recognition problem of video activities involving two interacting moving objects under a surveillance camera. We develop a novel video activity representation scheme --'bag of segments'. In this scheme, the video sessions are represented as a collection of independent segments, with memberships to each pre-learned visual patterns that we call codewords. To better represent the video segments with object interaction, we design a set of new features based on the prediction filter responses and the Granger Causality Test (GCT). These features capture the inter-relationship between moving objects and are combined with conventional features such as position and velocity. We validate the proposed method for the task of video activities classification with extensive experiments on a surveillance database with 867 video sessions.