A comprehensive representation scheme for video semantic ontology and its applications in semantic concept detection

Recent research has discovered that leveraging ontology is an effective way to facilitate semantic video concept detection. As an explicit knowledge representation, a formal ontology definition usually consists of a lexicon, properties, and relations. In this paper, we present a comprehensive representation scheme for video semantic ontology in which all the three components are well studied. Specifically, we leverage LSCOM to construct the concept lexicon, describe concept property as the weights of different modalities which are obtained manually or by data-driven approach, and model two types of concept relations (i.e., pairwise correlation and hierarchical relation). In contrast with most existing ontologies which are only focused on one or two components for domain-specific videos, the proposed ontology is more comprehensive and general. To validate the effectiveness of this ontology, we further apply it to video concept detection. The experiments on TRECVID 2005 corpus have demonstrated a superior performance compared to existing key approaches to video concept detection.

[1]  Steffen Staab,et al.  Semantic Annotation of Images and Videos for Multimedia Analysis , 2005, ESWC.

[2]  Alberto Del Bimbo,et al.  Video Annotation and Retrieval Using Ontologies and Rule Learning , 2010, IEEE MultiMedia.

[3]  Yiyu Yao,et al.  Information-Theoretic Measures for Knowledge Discovery and Data Mining , 2003 .

[4]  Jianping Fan,et al.  Building concept ontology for medical video annotation , 2006, MM '06.

[5]  John R. Smith,et al.  IBM Research TRECVID-2009 Video Retrieval System , 2009, TRECVID.

[6]  Yi Wu,et al.  Ontology-based multi-classification learning for video concept detection , 2004, 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763).

[7]  Ming-Syan Chen,et al.  Association and Temporal Rule Mining for Post-Filtering of Semantic Concept Detection in Video , 2008, IEEE Transactions on Multimedia.

[8]  Jawaharlal Karmeshu,et al.  Entropy Measures, Maximum Entropy Principle and Emerging Applications , 2003 .

[9]  John R. Smith,et al.  Multimedia semantic indexing using model vectors , 2003, 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698).

[10]  C. Tsinaraki,et al.  Interoperability Support for Ontology-Based Video Retrieval Applications , 2004, CIVR.

[11]  Shih-Fu Chang,et al.  Columbia University’s Baseline Detectors for 374 LSCOM Semantic Visual Concepts , 2007 .

[12]  Christiane Fellbaum,et al.  Book Reviews: WordNet: An Electronic Lexical Database , 1999, CL.

[13]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[14]  Christopher DeCoro,et al.  Bayesian Aggregation for Hierarchical Genre Classification , 2007, ISMIR.

[15]  Gaurav Harit,et al.  Using Multimedia Ontology for Generating Conceptual Annotations and Hyperlinks in Video Collections , 2006, 2006 IEEE/WIC/ACM International Conference on Web Intelligence (WI 2006 Main Conference Proceedings)(WI'06).

[16]  Marcel Worring,et al.  The challenge problem for automated detection of 101 semantic concepts in multimedia , 2006, MM '06.

[17]  Tao Mei,et al.  Building a comprehensive ontology to refine video concept detection , 2007, MIR '07.

[18]  M. Ashburner,et al.  Gene Ontology: tool for the unification of biology , 2000, Nature Genetics.

[19]  Marcel Worring,et al.  Concept-Based Video Retrieval , 2009, Found. Trends Inf. Retr..

[20]  T. Moon.  The expectation-maximization algorithm , 1996, IEEE Signal Process. Mag..

[21]  John R. Smith,et al.  Large-scale concept ontology for multimedia , 2006, IEEE MultiMedia.

[22]  Alberto Del Bimbo,et al.  Automatic video annotation using ontologies extended with visual information , 2005, MULTIMEDIA '05.

[23]  Jeff Z. Pan,et al.  Multimedia annotations on the semantic Web , 2006, IEEE Multimedia.

[24]  Alberto Del Bimbo,et al.  Dynamic Pictorially Enriched Ontologies for Digital Video Libraries , 2009, IEEE MultiMedia.