Semi-Automatic Tagging of Photo Albums via Exemplar Selection and Tag Inference

As one of the emerging Web 2.0 activities, tagging becomes a popular approach to manage personal media data, such as photo albums. A dilemma in tagging behavior is the users' manual efforts and the tagging accuracy: exhaustively tagging all photos in an album is labor-intensive and time-consuming, and simply entering tags for the whole album leads to unsatisfying results. In this paper, we propose a semi-automatic tagging scheme that aims to facilitate users in photo album tagging. The scheme is able to achieve a good trade-off between manual efforts and tagging accuracy as well as to adjust tagging performance according to the user's customization. For a given album, it first selects a set of representative exemplars for manual tagging via a temporally consistent affinity propagation algorithm, and the tags of the rest of the photos are automatically inferred. Then a constrained affinity propagation algorithm is applied to select a new set of exemplars for manual tagging in an incremental manner, based on which the performance of the tag inference in the previous round can be estimated. If the results are not satisfying enough, a further round of exemplar selection and tag inference will be implemented. This process repeats until satisfactory tagging results are achieved, and users can also stop the process at any time. Experimental results on real-world Flickr photo albums have demonstrated the effectiveness and usefulness of the proposed scheme.

[1]  B. S. Manjunath,et al.  Texture Features for Browsing and Retrieval of Image Data , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  Yuandong Tian,et al.  EasyAlbum: an interactive photo annotation system based on face clustering and re-ranking , 2007, CHI.

[3]  Roelof van Zwol,et al.  Flickr tag recommendation based on collective knowledge , 2008, WWW.

[4]  R. Manmatha,et al.  Automatic image annotation and retrieval using cross-media relevance models , 2003, SIGIR.

[5]  Edward Y. Chang,et al.  Support vector machine active learning for image retrieval , 2001, MULTIMEDIA '01.

[6]  Zoubin Ghahramani,et al.  Combining active learning and semi-supervised learning using Gaussian fields and harmonic functions , 2003, ICML 2003.

[7]  James Ze Wang,et al.  Real-Time Computerized Annotation of Pictures , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Dong Liu,et al.  Multiple-Instance Active Learning for Image Categorization , 2009, MMM.

[9]  Dong Liu,et al.  Tag ranking , 2009, WWW '09.

[10]  Markus A. Stricker,et al.  Similarity of color images , 1995, Electronic Imaging.

[11]  Benjamin B. Bederson,et al.  Semi-automatic photo annotation strategies using event based clustering and clothing based person recognition , 2007, Interact. Comput..

[12]  Xian-Sheng Hua,et al.  Finding image exemplars using fast sparse affinity propagation , 2008, ACM Multimedia.

[13]  Jitendra Malik,et al.  Normalized cuts and image segmentation , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[14]  Anil K. Jain,et al.  On image classification: city images vs. landscapes , 1998, Pattern Recognit..

[15]  Rong Yan,et al.  A learning-based hybrid tagging and browsing approach for efficient manual image annotation , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Nenghai Yu,et al.  Annotating personal albums via web mining , 2008, ACM Multimedia.

[17]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[18]  Ja-Ling Wu,et al.  SheepDog: group and tag recommendation for flickr photos by automatic search-based learning , 2008, ACM Multimedia.

[19]  Mor Naaman,et al.  Why we tag: motivations for annotation in mobile and online media , 2007, CHI.

[20]  Shih-Fu Chang,et al.  To search or to label?: predicting the performance of search-based automatic image classifiers , 2006, MIR '06.

[21]  Wei-Ta Chu,et al.  Automatic selection of representative photo and smart thumbnailing using near-duplicate detection , 2008, ACM Multimedia.

[22]  Delbert Dueck,et al.  Clustering by Passing Messages Between Data Points , 2007, Science.

[23]  Brendan J. Frey,et al.  A Binary Variable Model for Affinity Propagation , 2009, Neural Computation.

[24]  Jiebo Luo,et al.  Annotating photo collections by label propagation according to multiple similarity cues , 2008, ACM Multimedia.

[25]  Jiebo Luo,et al.  Image Annotation Within the Context of Personal Photo Collections Using Hierarchical Event and Scene Models , 2009, IEEE Transactions on Multimedia.

[26]  Shih-Fu Chang,et al.  Active Context-Based Concept Fusionwith Partial User Labels , 2006, 2006 International Conference on Image Processing.