Result diversification in image retrieval based on semantic distance

Abstract User requirements for result diversification in image retrieval have been increasing with the explosion of image resources. Result diversification requires that image retrieval systems are made capable of handling semantic gaps between image visual features and semantic concepts, and providing both relevant and diversified image results. Context information, such as captions, descriptions, and tags, provides opportunities for image retrieval systems to improve their result diversification. This study explores a mechanism for improving result diversification using the semantic distance of image social tags. We design and compare nine strategies that combine three different semantic distance algorithms (WordNet, Google Distance, and Explicit Semantic Analysis) with three re-ranking algorithms (MMR, xQuAD, and Score Difference) for result diversification. In order to better prove the effectiveness of our strategy of applying semantic information, we also make use of visual features of images for result diversification experiment and make comparison. Our data for experimentation were extracted from 269,648 images selected from the NUS-WIDE datasets with manually annotated subtopics. Experimental results affirm the effectiveness of applying semantic information for improving result diversification in image retrieval. In particular, WordNet-based semantic distance combined with the Score Difference (WordNet-DivScore) outperformed other strategies in diversifying image retrieval results.

[1]  Omar El Beqqali,et al.  Improving question answering systems by using the explicit semantic analysis method , 2016, 2016 11th International Conference on Intelligent Systems: Theories and Applications (SITA).

[2]  Christiane Fellbaum,et al.  Book Reviews: WordNet: An Electronic Lexical Database , 1999, CL.

[3]  Meng Wang,et al.  Tag-Based Social Image Search: Toward Relevant and Diverse Results , 2011, Social Media Modeling and Computing.

[4]  Steven C. H. Hoi,et al.  On Effective Personalized Music Retrieval by Exploring Online User Behaviors , 2016, SIGIR.

[5]  Mark Sanderson,et al.  Using score differences for search result diversification , 2014, SIGIR.

[6]  Nenghai Yu,et al.  Flickr distance , 2008, ACM Multimedia.

[7]  Bogdan Ionescu,et al.  Pseudo-relevance feedback diversification of social image retrieval results , 2017, Multimedia Tools and Applications.

[8]  Lin Wu,et al.  Max-sum diversification on image ranking with non-uniform matroid constraints , 2013, Neurocomputing.

[9]  Frédéric Jurie,et al.  Image re-ranking system based on closed frequent patterns , 2014, International Journal of Multimedia Information Retrieval.

[10]  Craig MacDonald,et al.  Explicit Search Result Diversification through Sub-queries , 2010, ECIR.

[11]  Jing Liu,et al.  Robust Structured Subspace Learning for Data Representation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Evgeniy Gabrilovich,et al.  Computing Semantic Relatedness Using Wikipedia-based Explicit Semantic Analysis , 2007, IJCAI.

[13]  Mohan S. Kankanhalli,et al.  Exploring User-Specific Information in Music Retrieval , 2017, SIGIR.

[14]  Prashant Upadhyay,et al.  Review of Content Based Image Retrieval Systems , 2015 .

[15]  Francesco G. B. De Natale,et al.  Multimodal Retrieval with Diversification and Relevance Feedback for Tourist Attraction Images , 2017, ACM Trans. Multim. Comput. Commun. Appl..

[16]  Tat-Seng Chua,et al.  Learning from Collective Intelligence , 2016, ACM Trans. Multim. Comput. Commun. Appl..

[17]  Francesco G. B. De Natale,et al.  A hybrid approach for retrieving diverse social images of landmarks , 2015, 2015 IEEE International Conference on Multimedia and Expo (ICME).

[18]  Paul M. B. Vitányi,et al.  The Google Similarity Distance , 2004, IEEE Transactions on Knowledge and Data Engineering.

[19]  Tat-Seng Chua,et al.  NUS-WIDE: a real-world web image database from National University of Singapore , 2009, CIVR '09.

[20]  Sourav S. Bhowmick,et al.  Tag-based social image retrieval: An empirical evaluation , 2011, J. Assoc. Inf. Sci. Technol..

[21]  Alberto Del Bimbo,et al.  Fisher Encoded Convolutional Bag-of-Windows for Efficient Image Retrieval and Social Image Tagging , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[22]  Meng Wang,et al.  Image Re-Ranking Based on Topic Diversity , 2017, IEEE Transactions on Image Processing.

[23]  Xueming Qian,et al.  Tag-Based Image Search by Social Re-ranking , 2016, IEEE Transactions on Multimedia.

[24]  Qi Tian,et al.  Personalized Social Image Recommendation Method Based on User-Image-Tag Model , 2017, IEEE Transactions on Multimedia.

[25]  Shih-Hsin Chen,et al.  A Content-Based Image Retrieval Method Based on the Google Cloud Vision API and WordNet , 2017, ACIIDS.

[26]  Bogdan Ionescu,et al.  Retrieving Diverse Social Images at MediaEval 2017: Challenges, Dataset and Evaluation , 2017, MediaEval.

[27]  Bogdan Ionescu,et al.  A relevance feedback perspective to image search result diversification , 2014, 2014 IEEE 10th International Conference on Intelligent Computer Communication and Processing (ICCP).

[28]  Dong Han,et al.  Sentiment Analysis of Micro-blog Integrated on Explicit Semantic Analysis Method , 2018, Wirel. Pers. Commun..

[29]  Ricardo da Silva Torres,et al.  Diversity-driven learning for multimodal image retrieval with relevance feedback , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[30]  Xueqi Cheng,et al.  Learning Maximal Marginal Relevance Model via Directly Optimizing Diversity Evaluation Measures , 2015, SIGIR.

[31]  Marcin Detyniecki,et al.  Using tree of concepts and hierarchical reordering for diversity in image retrieval , 2013, 2013 11th International Workshop on Content-Based Multimedia Indexing (CBMI).

[32]  Xuelong Li,et al.  Visual-Textual Joint Relevance Learning for Tag-Based Social Image Search , 2013, IEEE Transactions on Image Processing.

[33]  Takahiro Hara,et al.  Wikipedia-Based Semantic Similarity Measurements for Noisy Short Texts Using Extended Naive Bayes , 2015, IEEE Transactions on Emerging Topics in Computing.

[34]  Mark Sanderson,et al.  Diversity in Photo Retrieval: Overview of the ImageCLEFPhoto Task 2009 , 2009, CLEF.

[35]  Tetsuya Sakai,et al.  Search Result Diversification Based on Hierarchical Intents , 2015, CIKM.

[36]  Henning Müller,et al.  Div150Cred: A social image retrieval result diversification with user tagging credibility dataset , 2015, MMSys.

[37]  Yue Gao,et al.  Tag-based social image search with visual-text joint hypergraph learning , 2011, ACM Multimedia.

[38]  Jinhui Tang,et al.  Weakly Supervised Deep Metric Learning for Community-Contributed Image Retrieval , 2015, IEEE Transactions on Multimedia.

[39]  Adrian Iftene,et al.  Using Semantic Resources in Image Retrieval , 2016, KES.

[40]  Minglun Gong,et al.  CIDER: Concept-based image diversification, exploration, and retrieval , 2013, Inf. Process. Manag..

[41]  Jade Goldstein-Stewart,et al.  The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries , 1998, SIGIR Forum.

[42]  Katsumi Tanaka,et al.  Computing Tag-Diversity for Social Image Search , 2014, ICADL.

[43]  Craig MacDonald,et al.  Search Result Diversification , 2015, Found. Trends Inf. Retr..

[44]  Naif Alajlan,et al.  Exploiting visual saliency for increasing diversity of image retrieval results , 2015, Multimedia Tools and Applications.

[45]  Mohan S. Kankanhalli,et al.  ConTagNet: Exploiting User Context for Image Tag Recommendation , 2016, ACM Multimedia.

[46]  Hermann Ney,et al.  Jointly optimising relevance and diversity in image retrieval , 2009, CIVR '09.

[47]  Jinhui Tang,et al.  Weakly Supervised Deep Matrix Factorization for Social Image Understanding , 2017, IEEE Transactions on Image Processing.

[48]  Sourav S. Bhowmick,et al.  Quantifying tag representativeness of visual content of social images , 2010, ACM Multimedia.

[49]  Jian Zhang,et al.  Graph-based clustering and ranking for diversified image search , 2014, Multimedia Systems.