A document expansion framework for tag-based image retrieval

Purpose The purpose of this paper is to utilize document expansion techniques for improving image representation and retrieval. This paper proposes a concise framework for tag-based image retrieval (TBIR). Design/methodology/approach The proposed approach includes three core components: a strategy of selecting expansion (similar) images from the whole corpus (e.g. cluster-based or nearest neighbor-based); a technique for assessing image similarity, which is adopted for selecting expansion images (text, image, or mixed); and a model for matching the expanded image representation with the search query (merging or separate). Findings The results show that applying the proposed method yields significant improvements in effectiveness, and the method obtains better performance on the top of the rank and makes a great improvement on some topics with zero score in baseline. Moreover, nearest neighbor-based expansion strategy outperforms the cluster-based expansion strategy, and using image features for selecting expansion images is better than using text features in most cases, and the separate method for calculating the augmented probability P(q|RD) is able to erase the negative influences of error images in RD. Research limitations/implications Despite these methods only outperform on the top of the rank instead of the entire rank list, TBIR on mobile platforms still can benefit from this approach. Originality/value Unlike former studies addressing the sparsity, vocabulary mismatch, and tag relatedness in TBIR individually, the approach proposed by this paper addresses all these issues with a single document expansion framework. It is a comprehensive investigation of document expansion techniques in TBIR.

[1]  Wolfgang Nejdl,et al.  The Benefit of Using Tag-Based Profiles , 2007 .

[2]  Marco La Cascia,et al.  Unifying Textual and Visual Cues for Content-Based Image Retrieval on the World Wide Web , 1999, Comput. Vis. Image Underst..

[3]  Xian-Sheng Hua,et al.  Towards a Relevant and Diverse Search of Social Images , 2010, IEEE Transactions on Multimedia.

[4]  Vijay V. Raghavan,et al.  Content-Based Image Retrieval Systems - Guest Editors' Introduction , 1995, Computer.

[5]  Sourav S. Bhowmick,et al.  Social image tag recommendation by concept matching , 2011, ACM Multimedia.

[6]  Marcel Worring,et al.  Learning Social Tag Relevance by Neighbor Voting , 2009, IEEE Transactions on Multimedia.

[7]  Edward A. Fox,et al.  Combination of Multiple Searches , 1993, TREC.

[8]  S. Sclaroff,et al.  Combining textual and visual cues for content-based image retrieval on the World Wide Web , 1998, Proceedings. IEEE Workshop on Content-Based Access of Image and Video Libraries (Cat. No.98EX173).

[9]  Marcel Worring,et al.  Content-Based Image Retrieval at the End of the Early Years , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  Tat-Seng Chua,et al.  NUS-WIDE: a real-world web image database from National University of Singapore , 2009, CIVR '09.

[11]  Ivor W. Tsang,et al.  Tag-based web photo retrieval improved by batch mode re-tagging , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[12]  Sourav S. Bhowmick,et al.  Tag-based social image retrieval: An empirical evaluation , 2011, J. Assoc. Inf. Sci. Technol..

[13]  Xueming Qian,et al.  Tag-Based Image Search by Social Re-ranking , 2016, IEEE Transactions on Multimedia.

[14]  Antonio Torralba,et al.  Ieee Transactions on Pattern Analysis and Machine Intelligence 1 80 Million Tiny Images: a Large Dataset for Non-parametric Object and Scene Recognition , 2022 .

[15]  W. Bruce Croft,et al.  Cluster-based retrieval using language models , 2004, SIGIR '04.

[16]  Subhransu Maji,et al.  Automatic Image Annotation using Deep Learning Representations , 2015, ICMR.

[17]  Tao Tao,et al.  Language Model Information Retrieval with Document Expansion , 2006, NAACL.

[18]  Fabio Crestani,et al.  Tag data and personalized information retrieval , 2008, SSM '08.

[19]  Katrina Fenlon,et al.  Improving retrieval of short texts through document expansion , 2012, SIGIR '12.

[20]  Kristen Grauman,et al.  Kernelized locality-sensitive hashing for scalable image search , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[21]  Pinar Duygulu Sahin,et al.  Automatic tag expansion using visual similarity for photo sharing websites , 2010, Multimedia Tools and Applications.

[22]  Oded Nov,et al.  What drives content tagging: the case of photos on Flickr , 2008, CHI.

[23]  Shuicheng Yan,et al.  Inferring semantic concepts from community-contributed images and noisy tags , 2009, ACM Multimedia.

[24]  Brian D. Davison,et al.  Introduction to special section on adversarial issues in Web search , 2008, TWEB.

[25]  Dong Zhou,et al.  Document expansion for image retrieval , 2010, RIAO.

[26]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[27]  R. Manmatha,et al.  Automatic image annotation and retrieval using cross-media relevance models , 2003, SIGIR.

[28]  Dong Liu,et al.  Boost search relevance for tag-based social image retrieval , 2009, 2009 IEEE International Conference on Multimedia and Expo.

[29]  Ruixuan Li,et al.  Measuring Social Tag Confidence: Is It a Good or Bad Tag? , 2011, WAIM.

[30]  Xuelong Li,et al.  Visual-Textual Joint Relevance Learning for Tag-Based Social Image Search , 2013, IEEE Transactions on Image Processing.

[31]  Hong-Gee Kim,et al.  A social inverted index for social-tagging-based information retrieval , 2012, J. Inf. Sci..

[32]  Cordelia Schmid,et al.  Improving Bag-of-Features for Large Scale Image Search , 2010, International Journal of Computer Vision.

[33]  Mark S. Melenhorst,et al.  Tag-based information retrieval of video content , 2008, UXTV '08.

[34]  Shuicheng Yan,et al.  Image tag refinement towards low-rank, content-tag prior and error sparsity , 2010, ACM Multimedia.

[35]  Marcel Worring,et al.  Unsupervised multi-feature tag relevance learning for social image retrieval , 2010, CIVR '10.

[36]  W. Bruce Croft,et al.  LDA-based document models for ad-hoc retrieval , 2006, SIGIR.

[37]  Mark Sanderson,et al.  Test Collection Based Evaluation of Information Retrieval Systems , 2010, Found. Trends Inf. Retr..

[38]  Changsheng Xu,et al.  User-Aware Image Tag Refinement via Ternary Semantic Analysis , 2012, IEEE Transactions on Multimedia.

[39]  Georgia Koutrika,et al.  Combating spam in tagging systems: An evaluation , 2008, TWEB.

[40]  Alberto Del Bimbo,et al.  Image Tag Assignment, Refinement and Retrieval , 2015, ACM Multimedia.

[41]  Alberto Del Bimbo,et al.  Socializing the Semantic Gap , 2015, ACM Comput. Surv..

[42]  Sourav S. Bhowmick,et al.  Image tag clarity: in search of visual-representative tags for social images , 2009, WSM@MM.

[43]  Miles Efron,et al.  Hashtag retrieval in a microblogging environment , 2010, SIGIR.

[44]  Sourav S. Bhowmick,et al.  Content is still king: the effect of neighbor voting schemes on tag relevance for social image retrieval , 2012, ICMR.

[45]  Fei-Fei Li,et al.  Deep visual-semantic alignments for generating image descriptions , 2015, CVPR.

[46]  Lei Wu,et al.  Tag Completion for Image Retrieval , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Yong Yu,et al.  Optimizing web search using social annotations , 2007, WWW '07.

[48]  Xirong Li,et al.  Classifying tag relevance with relevant positive and negative examples , 2013, ACM Multimedia.

[49]  Yong Yu,et al.  Exploring folksonomy for personalized search , 2008, SIGIR '08.

[50]  Georgia Koutrika,et al.  Combating spam in tagging systems , 2007, AIRWeb '07.