Unsupervised Learning of Multi-Object Event Classes ∗

We present a novel approach for automatically inferring models of multiobject events. Objects are first detected and tracked, their motion is then segmented into a set of primitive events. These primitive events then form the nodes in a Markov network that encodes the entire event space. A bottomup/top-down search algorithm is developed to detect typical event structures that are used for classifying an observed multi-object event. We demonstrate our algorithm on clustering and inferring events in a table-laying scene.