Learning with structured data: applications to computer vision

In this thesis we address structured machine learning problems. Here “structured” refers to situations in which the input or output domain of a prediction function is non-vectorial. Instead, the input instance or the predicted value can be decomposed into parts that follow certain dependencies, relations and constraints. Throughout the thesis we will use hard computer vision tasks as a rich source of structured machine learning problems. In the first part of the thesis we consider structure in the input domain. We develop a general framework based on the notion of substructures. The framework is broadly applicable and we show how to cast two computer vision problems — class-level object recognition and human action recognition — in terms of classifying structured input data. For the class-level object recognition problem we model images as labeled graphs that encode local appearance statistics at vertices and pairwise geometric relations at edges. Recognizing an object can then be posed within our substructure framework as finding discriminative matching subgraphs. For the recognition of human actions we apply a similar principle in that we model a video as a sequence of local motion information. Recognizing an action then becomes recognizing a matching subsequence within the larger video sequence. For both applications, our framework enables us to finding the discriminative substructures from training data. This first part contains as a main contribution a set of abstract algorithms for our framework to enable the construction of powerful classifiers for a large family of structured input domains. The second part of the thesis addresses structure in the output domain of a prediction function. Specifically we consider image segmentation problems in which the produced segmentation must satisfy global properties such as connectivity. We develop a principled method to incorporate global interactions into computer vision random field models by means of linear programming relaxations. To further understand solutions produced by general linear programming relaxations we develop a tractable and novel concept of solution stability, where stability is quantified with respect to perturbations of the input data. This second part of the thesis makes progress in modeling, solving and understanding solution properties of hard structured prediction problems arising in computer vision. In particular, we show how previously intractable models integrating global constraints with local evidence can be well approximated. We further show how these solutions can be understood in light of their stability properties.