Sketch-Based Image Retrieval by Salient Contour Reinforcement

The paper presents a sketch-based image retrieval algorithm. One of the main challenges in sketch-based image retrieval (SBIR) is to measure the similarity between a sketch and an image. To tackle this problem, we propose an SBIR-based approach by salient contour reinforcement. In our approach, we divide the image contour into two types. The first is the global contour map. The second, called the salient contour map, is helpful to find out the object in images similar to the query. In addition, based on the two contour maps, we propose a new descriptor, namely an angular radial orientation partitioning (AROP) feature. It fully utilizes the edge pixels' orientation information in contour maps to identify the spatial relationships. Our AROP feature based on the two candidate contour maps is both efficient and effective to discover false matches of local features between sketches and images, and can greatly improve the retrieval performance. The application of the retrieval system based on this algorithm is established. The experiments on the image dataset with 0.3 million images show the effectiveness of the proposed method and comparisons with other algorithms are also given. Compared to baseline performance, the proposed method achieves 10% higher precision in top 5.

[1]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Toshikazu Kato,et al.  Query by Visual Example - Content based Image Retrieval , 1992, EDBT.

[3]  David Salesin,et al.  Fast multiresolution image querying , 1995, SIGGRAPH.

[4]  Anil K. Jain,et al.  Image retrieval using color and shape , 1996, Pattern Recognit..

[5]  Shih-Fu Chang,et al.  VisualSEEk: a fully automated content-based image query system , 1997, MULTIMEDIA '96.

[6]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[7]  Jau-Ling Shih,et al.  A new system for trademark segmentation and retrieval , 2001, Image Vis. Comput..

[8]  Jitendra Malik,et al.  Learning to detect natural image boundaries using local brightness, color, and texture cues , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Joseph J. LaViola,et al.  MathPad2: a system for the creation and exploration of mathematical sketches , 2004, SIGGRAPH 2004.

[10]  A. Chalechale,et al.  Edge image description using angular radial partitioning , 2004 .

[11]  Abdolah Chalechale,et al.  Sketch-based image matching Using Angular partitioning , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[12]  Xiaolin Wu,et al.  Directly operable image representation of multiscale primal sketch , 2005, IEEE Trans. Multim..

[13]  Randall Davis,et al.  LADDER, a sketching language for user interface developers , 2005, Comput. Graph..

[14]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[15]  Björn Stenger,et al.  Model-based hand tracking using a hierarchical Bayesian filter , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  R. Davis,et al.  Sketch based interfaces: early processing for sketch understanding , 2007, SIGGRAPH Courses.

[17]  Joseph J. LaViola,et al.  MathPad2: a system for the creation and exploration of mathematical sketches , 2004, ACM Trans. Graph..

[18]  Marc Alexa,et al.  A descriptor for large scale image retrieval based on sketched feature lines , 2009, SBIM '09.

[19]  Shi-Min Hu,et al.  Sketch2Photo: internet image montage , 2009, ACM Trans. Graph..

[20]  Daniel Dixon,et al.  iCanDraw: using sketch recognition and corrective feedback to assist a user in drawing human faces , 2010, CHI.

[21]  Marc Alexa,et al.  An evaluation of descriptors for large-scale image retrieval from sketched feature lines , 2010, Comput. Graph..

[22]  Liqing Zhang,et al.  MindFinder: interactive sketch-based image search on millions of images , 2010, ACM Multimedia.

[23]  Rui Hu,et al.  Gradient field descriptor for sketch based retrieval and localization , 2010, 2010 IEEE International Conference on Image Processing.

[24]  Jitendra Malik,et al.  Shape matching and object recognition using shape contexts , 2010, 2010 3rd International Conference on Computer Science and Information Technology.

[25]  Liqing Zhang,et al.  Edgel index for large-scale sketch-based image search , 2011, CVPR 2011.

[26]  Yong Jae Lee,et al.  ShadowDraw: real-time user guidance for freehand drawing , 2011, ACM Trans. Graph..

[27]  Marc Alexa,et al.  Sketch-Based Image Retrieval: Benchmark and Bag-of-Features Descriptors , 2011, IEEE Transactions on Visualization and Computer Graphics.

[28]  Shi-Min Hu,et al.  Global contrast based salient region detection , 2011, CVPR 2011.

[29]  Matthijs C. Dorst Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[30]  Randall Davis,et al.  ChemInk: a natural real-time recognition system for chemical drawings , 2011, IUI '11.

[31]  Winston H. Hsu,et al.  Sketch-based image retrieval on mobile devices using compact hash bits , 2012, ACM Multimedia.

[32]  Marc Alexa,et al.  How do humans sketch objects? , 2012, ACM Trans. Graph..

[33]  Nilgun Donmez,et al.  Concepture: a regular language based framework for recognizing gestures with varying and repetitive patterns , 2012, SBIM '12.

[34]  Liqing Zhang,et al.  Sketch-based image retrieval on a large scale database , 2012, ACM Multimedia.

[35]  Xianming Liu,et al.  The scale of edges , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  Xiaochun Cao,et al.  SYM-FISH: A Symmetry-Aware Flip Invariant Sketch Histogram Shape Descriptor , 2013, 2013 IEEE International Conference on Computer Vision.

[37]  Lei Luo,et al.  Shape Similarity Analysis by Self-Tuning Locally Constrained Mixed-Diffusion , 2013, IEEE Transactions on Multimedia.

[38]  Lei Guo,et al.  An Object-Oriented Visual Saliency Detection Framework Based on Sparse Coding Representations , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[39]  Shi-Min Hu,et al.  PoseShop: Human Image Database Construction and Personalized Content Synthesis , 2013, IEEE Transactions on Visualization and Computer Graphics.

[40]  Yuan Yan Tang,et al.  GPS Estimation for Places of Interest From Social Users' Uploaded Photos , 2013, IEEE Transactions on Multimedia.

[41]  Qi Tian,et al.  Multimedia search reranking: A literature survey , 2014, CSUR.

[42]  Tinne Tuytelaars,et al.  Sketch classification and classification-driven analysis using Fisher vectors , 2014, ACM Trans. Graph..

[43]  Junwei Han,et al.  Efficient, simultaneous detection of multi-class geospatial targets based on visual saliency modeling and discriminative learning of sparse coding , 2014 .

[44]  Ping Shi,et al.  Sketch-based image retrieval by using saliency , 2014, 2014 11th International Conference on Fuzzy Systems and Knowledge Discovery (FSKD).

[45]  Antonio Marín-Hernández,et al.  Garabato: A proposal of a sketch-based Image Retrieval system for the Web , 2014, CONIELECOMP.

[46]  Cheng Jin,et al.  Sketch-Based Image Retrieval via Adaptive Weighting , 2014, ICMR.

[47]  Yuan Yan Tang,et al.  Landmark Summarization With Diverse Viewpoints , 2015, IEEE Transactions on Circuits and Systems for Video Technology.

[48]  Yichen Wei,et al.  Sketch Matching on Topology Product Graph , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[49]  Xueming Qian,et al.  Scalable Mobile Image Retrieval by Exploring Contextual Saliency , 2015, IEEE Transactions on Image Processing.

[50]  Xueming Qian,et al.  Image Location Estimation by Salient Region Matching , 2015, IEEE Transactions on Image Processing.

[51]  钱学明 Sketch-based Image Retrieval Using Contour Segments , 2015 .

[52]  Shu Wang,et al.  Sketch-Based Image Retrieval Through Hypothesis-Driven Object Boundary Selection With HLR Descriptor , 2015, IEEE Transactions on Multimedia.

[53]  Stefano Soatto,et al.  Shape Matching Using Multiscale Integral Invariants , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[54]  Ling Shao,et al.  Learning Computational Models of Video Memorability from fMRI Brain Imaging , 2015, IEEE Transactions on Cybernetics.

[55]  Feng Wu,et al.  Background Prior-Based Salient Object Detection via Deep Reconstruction Residual , 2015, IEEE Transactions on Circuits and Systems for Video Technology.

[56]  Steven C. H. Hoi,et al.  Fast Object Retrieval Using Direct Spatial Matching , 2015, IEEE Transactions on Multimedia.

[57]  Meng Wang,et al.  Enhancing Sketch-Based Image Retrieval by Re-Ranking and Relevance Feedback , 2016, IEEE Transactions on Image Processing.

[58]  Xuelong Li,et al.  Two-Stage Learning to Predict Human Eye Fixations via SDAEs , 2016, IEEE Transactions on Cybernetics.