Incorporating concept ontology to enable probabilistic concept reasoning for multi-level image annotation

To enable automatic multi-level image annotation, we have addressed two inter-related important issues:(1)more effective framework for image content representation and feature extraction to characterize the middle-level semantics of image contents;(2)new framework for hierarchical probabilistic image concept reasoning and detection. To address the first issue salient objects are used as the semantic building blocks to characterize the middle-level semantics of image contents effectively while reducing the image analysis cost significantly. We have proposed three approaches to designing the detection functions for automatic salient object detection,and automatic function selection is also supported to find the "right "assumptions of the principal visual properties for the corresponding salient object classes. To address the second issue wehaveproposed a novel framework to incorporate the concept ontology to achieve hierarchical probabilistic image concept reasoning for multi-level image annotation. The concept ontology for a large-scale public image database called Label Me is semi-automatically derived from the available image labels by using WordNet The image concepts at the first level of the concept ontology are used to characterize the most specific semantics of image contents with the smallest variations, and their correspondences with the semantic building blocks (i.e.,salient objects)are well-de fined and can be modeled accurately by using Bayesian networks. In addition,the predictions of the appearances of the higher-level image concepts with large variations are adopted by the underlying concept ontology or by combining the available predictions of the appearances of their children concepts through hierarchical Bayesian networks.Our experiments on a large public dataset have shown that our framework for hierarchical probabilistic image concept reasoning is scalable to diverse image contents (i.e.,large amount of salient object classes)with large within-category variations.

[1]  Jianping Fan,et al.  Mining images on semantics via statistical learning , 2005, KDD '05.

[2]  Daniel Gatica-Perez,et al.  PLSA-based image auto-annotation: constraining the latent space , 2004, MULTIMEDIA '04.

[3]  Richard E. Neapolitan,et al.  Learning Bayesian networks , 2007, KDD '07.

[4]  James Ze Wang,et al.  SIMPLIcity: Semantics-Sensitive Integrated Matching for Picture LIbraries , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Christiane Fellbaum,et al.  Book Reviews: WordNet: An Electronic Lexical Database , 1999, CL.

[6]  Nuno Vasconcelos,et al.  Image indexing with mixture hierarchies , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[7]  Susan T. Dumais,et al.  Hierarchical classification of Web content , 2000, SIGIR '00.

[8]  Alexander G. Hauptmann,et al.  Towards a Large Scale Concept Ontology for Broadcast Video , 2004, CIVR.

[9]  W. Bruce Croft,et al.  Deriving concept hierarchies from text , 1999, SIGIR '99.

[10]  Marcel Worring,et al.  Content-Based Image Retrieval at the End of the Early Years , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[11]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[12]  W. Bruce Croft,et al.  Generating hierarchical summaries for web searches , 2003, SIGIR '03.

[13]  Wei-Ying Ma,et al.  Learning and inferring a semantic space from user's relevance feedback for image retrieval , 2002, MULTIMEDIA '02.

[14]  John R. Smith,et al.  Semantic representation: search and mining of multimedia content , 2004, KDD '04.

[15]  James Ze Wang,et al.  Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[16]  John R. Smith,et al.  Image Classification and Querying Using Composite Region Templates , 1999, Comput. Vis. Image Underst..

[17]  David A. Forsyth,et al.  Learning the semantics of words and pictures , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[18]  Dorin Comaniciu,et al.  Mean Shift: A Robust Approach Toward Feature Space Analysis , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  Jianping Fan,et al.  Multi-level annotation of natural scenes using dominant image components and semantic concepts , 2004, MULTIMEDIA '04.

[20]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[21]  Rainer Lienhart,et al.  Classifying images on the web automatically , 2002, J. Electronic Imaging.

[22]  David Maxwell Chickering,et al.  Learning Bayesian Networks: The Combination of Knowledge and Statistical Data , 1994, Machine Learning.

[23]  Pietro Perona,et al.  A Bayesian approach to unsupervised one-shot learning of object categories , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[24]  Shih-Fu Chang,et al.  Image classification using multimedia knowledge networks , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[25]  Antonio Torralba,et al.  Sharing features: efficient boosting procedures for multiclass object detection , 2004, CVPR 2004.

[26]  Clement T. Yu,et al.  Evaluating strategies and systems for content based indexing of person images on the Web , 2000, ACM Multimedia.

[27]  Shih-Fu Chang,et al.  MediaNet: a multimedia information network for knowledge representation , 2000, SPIE Optics East.

[28]  Jing Huang,et al.  An automatic hierarchical image classification scheme , 1998, MULTIMEDIA '98.

[29]  Jianping Fan,et al.  Automatic image annotation by incorporating feature hierarchy and boosting to scale up SVM classifiers , 2006, MM '06.

[30]  Kien A. Hua,et al.  Image Retrieval Based on Regions of Interest , 2003, IEEE Trans. Knowl. Data Eng..

[31]  Jitendra Malik,et al.  Blobworld: Image Segmentation Using Expectation-Maximization and Its Application to Image Querying , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  Jianping Fan,et al.  Semantic image classification with hierarchical feature subset selection , 2005, MIR '05.

[33]  Jiebo Luo,et al.  Improved scene classification using efficient low-level features and semantic cues , 2004, Pattern Recognit..

[34]  John Platt,et al.  Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods , 1999 .

[35]  Rong Jin,et al.  Using a probabilistic source model for comparing images , 2002, Proceedings. International Conference on Image Processing.

[36]  Daniel Gatica-Perez,et al.  On image auto-annotation with latent space models , 2003, ACM Multimedia.

[37]  Shih-Fu Chang,et al.  Image Retrieval: Current Techniques, Promising Directions, and Open Issues , 1999, J. Vis. Commun. Image Represent..

[38]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[39]  Anil K. Jain,et al.  Image classification for content-based indexing , 2001, IEEE Trans. Image Process..

[40]  William I. Grosky,et al.  Negotiating the semantic gap: from feature maps to semantic landscapes , 2001, Pattern Recognit..

[41]  Thomas Hofmann,et al.  Text classification in a hierarchical mixture model for small training sets , 2001, CIKM '01.

[42]  Shih-Fu Chang,et al.  Visual information retrieval from large distributed online repositories , 1997, CACM.

[43]  Yihong Gong.  Advancing content-based image retrieval by exploiting image color and region features , 1999, Multimedia Systems.