Global structured models towards scene understanding

Many scene understanding tasks are formulated as a labelling problem that tries to assign a label to each pixel of an image. These discrete labels may vary depending on the task, for example they may correspond to di erent object classes such as car, grass or sky, or to depths or to intensity after denoising. These labelling problems are typically formulated as a pairwise Markov or Conditional Random Field, modelling the dependencies of labels of pairs of variables in the local neighbourhoods. However, these pairwise models are very restricted in their expressivity. They can not model rich natural statistics and induce desired complex structures in the output labelling. In this thesis we propose global structured formulations beyond pairwise models, showing that they are very useful in computer vision, furthermore that they can still be learnt and optimised e ciently. First we propose a model, which generalises existing approaches for semantic object class segmentation, formulated in terms of pixels, segments or groups of segments. The proposed method e ciently integrates the strengths of these di erent approaches, capturing discriminative information across di erent scales. Next we show how the standard approaches for the semantic object class segmentation problem can be improved by the inclusion of costs based on high level statistics, including object class co-occurrence, which capture knowledge of scene semantics, for example that motorbikes and cows are unlikely to occur together in an image. Then we propose a novel latent random eld support vector machine for object detection with a convex mrf regularization and suggest a way to include this information in the object class segmentation formulation. Finally we propose a model that jointly estimates labellings of multiple domains over a product space of labels. We demonstrate the usefulness of this model on the problem of joint object class semantic segmentation and dense 3D stereo reconstruction and show that this approach signi cantly outperforms existing methods. We show that all proposed models can be optimised e ciently using powerful graph cut based move making algorithms. List of Publications Journals ‰ubor Ladický, Chris Russell, Pushmeet Kohli, Philip H.S. Torr Inference Methods for CRFs with Co-occurrence Statistics International Journal of Computer Vision, 2011 Invited Paper ‰ubor Ladický, Paul Sturgess, Chris Russell, Sunando Sengupta, Yalin Bastanlar, William Clocksin, Philip H.S. Torr Joint Optimisation for Object Class Segmentation and Dense Stereo Reconstruction International Journal of Computer Vision, 2011 Invited Paper Pushmeet Kohli, ‰ubor Ladický, Philip H.S. Torr Robust Higher Order Potentials for Enforcing Label Consistency International Journal of Computer Vision, 2009 Conferences ‰ubor Ladický, Philip H.S. Torr Locally Linear Support Vector Machines International Conference on Machine Learning, 2011 ‰ubor Ladický, Chris Russell, Pushmeet Kohli, Philip H.S. Torr Graph Cut based Inference with Co-occurrence Statistics European Conference on Computer Vision, 2010 Best Paper Award ‰ubor Ladický, Paul Sturgess, Karteek Alahari, Chris Russell, Philip H.S. Torr What,Where & How Many? Combining Object Detectors and CRFs European Conference on Computer Vision, 2010 ‰ubor Ladický, Paul Sturgess, Chris Russell, Sunando Sengupta, Yalin Bastanlar, William Clocksin, Philip H.S. Torr Joint Optimisation for Object Class Segmentation and Dense Stereo Reconstruction British Machine Vision Conference, 2010 Best Paper Award Chris Russell, ‰ubor Ladický, Pushmeet Kohli, Philip H.S. Torr Exact and Approximate Inference in Associative Hierarchical Networks using GraphCuts Conference on Uncertainty in Arti cial Intelligence, 2010 ‰ubor Ladický, Chris Russell, Pushmeet Kohli, Philip H.S. Torr Associative Hierarchical CRFs for Object Class Image Segmentation International Conference on Computer Vision, 2009 Paul Sturgess, Karteek Alahari, ‰ubor Ladický, Philip H.S. Torr Combining Appearance and Structure from Motion Features for Road Scene Understanding British Machine Vision Conference, 2009 Pushmeet Kohli, ‰ubor Ladický, Philip H.S. Torr Robust Higher Order Potentials for Enforcing Label Consistency Conference on Computer Vision and Pattern Recognition, 2008