Enhancing Sketch-Based Image Retrieval by Re-Ranking and Relevance Feedback

A sketch-based image retrieval often needs to optimize the tradeoff between efficiency and precision. Index structures are typically applied to large-scale databases to realize efficient retrievals. However, the performance can be affected by quantization errors. Moreover, the ambiguousness of user-provided examples may also degrade the performance, when compared with traditional image retrieval methods. Sketch-based image retrieval systems that preserve the index structure are challenging. In this paper, we propose an effective sketch-based image retrieval approach with re-ranking and relevance feedback schemes. Our approach makes full use of the semantics in query sketches and the top ranked images of the initial results. We also apply relevance feedback to find more relevant images for the input query sketch. The integration of the two schemes results in mutual benefits and improves the performance of the sketch-based image retrieval.

[1]  Marc Alexa,et al.  Sketch-Based Image Retrieval: Benchmark and Bag-of-Features Descriptors , 2011, IEEE Transactions on Visualization and Computer Graphics.

[2]  Gerard Salton,et al.  Improving retrieval performance by relevance feedback , 1997, J. Am. Soc. Inf. Sci..

[3]  David Nistér,et al.  Scalable Recognition with a Vocabulary Tree , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[4]  A. Chalechale,et al.  Edge image description using angular radial partitioning , 2004 .

[5]  Jiri Matas,et al.  Total recall II: Query expansion revisited , 2011, CVPR 2011.

[6]  James Ze Wang,et al.  Image retrieval: Ideas, influences, and trends of the new age , 2008, CSUR.

[7]  Giorgos Stamou,et al.  Context-sensitive semantic query expansion , 2002, Proceedings 2002 IEEE International Conference on Artificial Intelligence Systems (ICAIS 2002).

[8]  Ximena Olivares,et al.  Visual diversification of image search results , 2009, WWW '09.

[9]  Meng Wang,et al.  Multimodal Graph-Based Reranking for Web Image Search , 2012, IEEE Transactions on Image Processing.

[10]  Ferran Marqués,et al.  Region-based representations of image and video: segmentation tools for multimedia services , 1999, IEEE Trans. Circuits Syst. Video Technol..

[11]  Preecha Sangassapaviriya,et al.  Feature extraction for content-based image retrieval , 1999 .

[12]  J. Jaafar,et al.  Exploiting short query expansion for images retrieval , 2012, 2012 International Conference on Computer & Information Science (ICCIS).

[13]  Ming Yang,et al.  Contextual weighting for vocabulary tree based image retrieval , 2011, 2011 International Conference on Computer Vision.

[14]  S. H. Srinivasan,et al.  Finding near-duplicate images on the web using fingerprints , 2008, ACM Multimedia.

[15]  Yuan Yan Tang,et al.  GPS Estimation for Places of Interest From Social Users' Uploaded Photos , 2013, IEEE Transactions on Multimedia.

[16]  Ingemar J. Cox,et al.  The Bayesian image retrieval system, PicHunter: theory, implementation, and psychophysical experiments , 2000, IEEE Trans. Image Process..

[17]  Toshikazu Kato,et al.  Query by Visual Example - Content based Image Retrieval , 1992, EDBT.

[18]  Shi-Min Hu,et al.  Sketch2Photo: internet image montage , 2009, ACM Trans. Graph..

[19]  Liqing Zhang,et al.  Edgel index for large-scale sketch-based image search , 2011, CVPR 2011.

[20]  Shuang Liang,et al.  Sketch retrieval and relevance feedback with biased SVM classification , 2008, Pattern Recognit. Lett..

[21]  Raghu Machiraju,et al.  Geometric verification of swirling features in flow fields , 2002, IEEE Visualization, 2002. VIS 2002..

[22]  Bipin C. Desai,et al.  Visual Keyword-based Image Retrieval using Latent Semantic Indexing, Correlation-enhanced Similarity Matching and Query Expansion in Inverted Index , 2006, 2006 10th International Database Engineering and Applications Symposium (IDEAS'06).

[23]  Thomas S. Huang,et al.  Relevance feedback: a power tool for interactive content-based image retrieval , 1998, IEEE Trans. Circuits Syst. Video Technol..

[24]  Thomas S. Huang,et al.  Content-based image retrieval with relevance feedback in MARS , 1997, Proceedings of International Conference on Image Processing.

[25]  Raimondo Schettini,et al.  Feature Extraction for Content-Based Image Retrieval , 2009, Encyclopedia of Database Systems.

[26]  Wei-Ying Ma,et al.  Hierarchical clustering of WWW image search results using visual, textual and link information , 2004, MULTIMEDIA '04.

[27]  Yuting Zhang,et al.  Sketch-based image retrieval using contour segments , 2015, 2015 IEEE 17th International Workshop on Multimedia Signal Processing (MMSP).

[28]  Seiji Yamada,et al.  Semisupervised Query Expansion with Minimal Feedback , 2007, IEEE Transactions on Knowledge and Data Engineering.

[29]  Raimondo Schettini,et al.  Feature Extraction for Content-Based Image Retrieval , 2009, Encyclopedia of Database Systems.

[30]  Michael Isard,et al.  Lost in quantization: Improving particular object retrieval in large scale image databases , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[31]  Jitendra Malik,et al.  Learning to detect natural image boundaries using local brightness, color, and texture cues , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Michael Isard,et al.  Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[33]  Tao Mei,et al.  Contextual Video Recommendation by Multimodal Relevance and User Feedback , 2011, TOIS.

[34]  Lei Zhang,et al.  A Unified Relevance Feedback Framework for Web Image Retrieval , 2009, IEEE Transactions on Image Processing.

[35]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[36]  Qi Jia,et al.  Query by sketch: An asymmetric sketch-vs-image retrieval system , 2011, 2011 4th International Congress on Image and Signal Processing.

[37]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[38]  Shiliang Zhang,et al.  Edge-SIFT: Discriminative Binary Descriptor for Scalable Partial-Duplicate Mobile Search , 2013, IEEE Transactions on Image Processing.

[39]  Dong Wang,et al.  Robust semantic sketch based specific image retrieval , 2010, 2010 IEEE International Conference on Multimedia and Expo.

[40]  Alireza Khotanzad,et al.  Invariant Image Recognition by Zernike Moments , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[41]  S. Yadav,et al.  Sketch4Match – Content-based Image Retrieval System Using Sketches , 2012 .

[42]  Chong-Wah Ngo,et al.  Scale-Rotation Invariant Pattern Entropy for Keypoint-Based Near-Duplicate Detection , 2009, IEEE Transactions on Image Processing.

[43]  Eugenio Di Sciascio,et al.  Content-Based Image Retrieval over the Web Using Query by Sketch and Relevance Feedback , 1999, VISUAL.

[44]  Björn Stenger,et al.  Model-based hand tracking using a hierarchical Bayesian filter , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[45]  Geoff Wyvill,et al.  SIFT and SURF Performance Evaluation against Various Image Deformations on Benchmark Dataset , 2011, 2011 International Conference on Digital Image Computing: Techniques and Applications.

[46]  Xueming Qian,et al.  Image Location Estimation by Salient Region Matching , 2015, IEEE Transactions on Image Processing.

[47]  Abdolah Chalechale,et al.  Sketch-based image matching Using Angular partitioning , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[48]  Yuan Yan Tang,et al.  GPS Estimation from Users' Photos , 2013, MMM.

[49]  Xueming Qian,et al.  Scalable Mobile Image Retrieval by Exploring Contextual Saliency , 2015, IEEE Transactions on Image Processing.

[50]  Tat-Seng Chua,et al.  Relevance feedback techniques for color-based image retrieval , 1998, Proceedings 1998 MultiMedia Modeling. MMM'98 (Cat. No.98EX200).