Collaborative and content-based image labeling

Many on-line photo sharing systems allow users to tag their images so as to support semantic image search. In this paper, we study how one can take advantages of the already-tagged images to (semi-)automate the labeling of newly uploaded ones. In particular, we propose a hybrid approach for the prediction where user-provided tags and image visual contents are fused under a unified probabilistic framework. Kernel smoothing and collaborative filtering techniques are explored for improving the accuracy of the probabilistic models estimation. By comparing with some state-of-the-art content-based image labeling methods, we have empirically shown that 1) the proposed method can achieve comparable tag prediction accuracy when there is no user-provided tag, and that 2) it can significantly boost the prediction accuracy if the user can provide just a few tags.

[1]  John Riedl,et al.  An algorithmic framework for performing collaborative filtering , 1999, SIGIR '99.

[2]  David M. Pennock,et al.  Categories and Subject Descriptors , 2001 .

[3]  R. Manmatha,et al.  A Model for Learning the Semantics of Pictures , 2003, NIPS.

[4]  Takeo Kanade,et al.  Content-Free Image Retrieval by Combinations of Keywords and User Feedbacks , 2005, CIVR.

[5]  G. Qiu.  Indexing chromatic and achromatic patterns for content-based colour image retrieval , 2002, Pattern Recognit..

[6]  David A. Forsyth,et al.  Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary , 2002, ECCV.

[7]  Mary Czerwinski,et al.  Semi-Automatic Image Annotation , 2001, INTERACT.

[8]  Gustavo Carneiro,et al.  Supervised Learning of Semantic Classes for Image Annotation and Retrieval , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  David Heckerman,et al.  Empirical Analysis of Predictive Algorithms for Collaborative Filtering , 1998, UAI.

[10]  E. Parzen.  On Estimation of a Probability Density Function and Mode , 1962 .

[11]  Latifur Khan,et al.  Image annotations by combining multiple evidence & wordNet , 2005, ACM Multimedia.

[12]  Sholom M. Weiss,et al.  Lightweight Collaborative Filtering Method for Binary-Encoded Data , 2001, PKDD.

[13]  Bijan Parsia,et al.  PhotoStuff-An Image Annotation Tool for the Semantic Web , 2005 .

[14]  Marcel Worring,et al.  Content-Based Image Retrieval at the End of the Early Years , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  R. Manmatha,et al.  Multiple Bernoulli relevance models for image and video annotation , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..