A Hybrid Framework for Detecting the Semantics of Concepts and Context

Semantic understanding of multimedia content necessitates models for the semantics of concepts, context and structure. We propose a hybrid framework that can combine discriminant or generative models for concepts with generative models for structure and context. Using the TREC Video 2002 benchmark corpus we show that robust models can be built for several diverse visual semantic concepts. We use a novel factor graphical framework to model inter-conceptual context for 12 semantic concepts of the corpus. Using the sum-product algorithm [1] for approximate or exact inference in these factor graph multinets, we attempt to correct errors made during isolated concept detection by forcing high-level constraints. This results in a significant improvement in the overall detection performance. Enforcement of this probabilistic context model enhances the detection performance further to 22% using the global multinet, whereas its factored approximation also leads to improvement by 18% over the baseline concept detection. This improvement is achieved without using any additional training data or separate annotations.