Multi-state Discriminative Video Segment Selection for Complex Event Classification

Recognizing long range complex events composed of a sequence of primitive actions is a challenging task, as videos may not be consistently aligned in time with respect to the primitive actions across video examples. Moreover, there can exist arbitrary long intervals between, and within the execution of primitive actions. We propose a novel multistate segment selection algorithm, which pools features from the discriminative segments of a video. We present an efficient linear programming based solution, and introduce novel linear constraints to enforce temporal ordering between segments from different states. We also propose a regularized version of our algorithm, which automatically determines the optimum number of segments to be selected in each video. Furthermore, we present a new, provably faster \(O(N\log N)\) algorithm for the single state K-segment selection problem. Our results are validated on the Composite Cooking Activity dataset, containing videos of cooking recipes.