Recognizing high-level audio-visual concepts using context

The recognition of high-level semantics from audio-visual data is a challenging multimedia understanding problem. The difficulty mainly lies in the gap that exists between low level media features and high level semantic concepts. In an attempt to bridge this gap we proposed a probabilistic framework for semantic understanding. The components of this framework are probabilistic multimedia objects and a graphical network of such objects. We show how the framework supports detection of multiple high-level concepts, which enjoy spatial and temporal support. More importantly, we show why context matters and how it can be modeled. Using a factor graph framework, we model context and use it to improve the detection of sites, objects and events. Using the concepts outdoor and a flying-helicopter we demonstrate how the factor graph multinet models context. Using ROC curves and probability of error curves we support the intuition that context should help.