A survey of sketch-based image retrieval

Sketch-based image retrieval (SBIR) has been studied since the early 1990s and has drawn more and more interest recently. Yet, a comprehensive review of the SBIR field is still absent. This survey tries to fill in this gap by reviewing the representative papers studying the SBIR problem. More importantly, this survey tries to answer two important questions which are generally not well discussed: what are the objectives of SBIR, and what is the general methodology of SBIR? The reviewed papers are organized in a chronological way and analyzed by answering these two important questions. As a novel trend, fine-grained SBIR has become the main topic for the recent research. The discussion on it is also integrated. From this survey, we hope that different perspectives can be observed, common values can be discovered and new ideas can be inspired.

[1]  Li.  Fine-grained sketch-based image retrieval by matching deformable part models , 2014 .

[2]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[3]  K. Wakimoto,et al.  Efficient and Effective Querying by Image Content , 1994 .

[4]  Alberto Del Bimbo,et al.  Image retrieval by elastic matching of shapes and image patterns , 1996, Proceedings of the Third IEEE International Conference on Multimedia Computing and Systems.

[5]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Marc Alexa,et al.  A descriptor for large scale image retrieval based on sketched feature lines , 2009, SBIM '09.

[7]  ZissermanAndrew,et al.  The Pascal Visual Object Classes Challenge , 2015 .

[8]  Anurag Mittal,et al.  Similarity-Invariant Sketch-Based Image Retrieval in Large Databases , 2014, ECCV.

[9]  Björn Stenger,et al.  Shape context and chamfer matching in cluttered scenes , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[10]  Keinosuke Fukunaga,et al.  A Branch and Bound Algorithm for Computing k-Nearest Neighbors , 1975, IEEE Transactions on Computers.

[11]  Jitendra Malik,et al.  Shape matching and object recognition using shape contexts , 2010, 2010 3rd International Conference on Computer Science and Information Technology.

[12]  Marc Alexa,et al.  Sketch-Based Image Retrieval: Benchmark and Bag-of-Features Descriptors , 2011, IEEE Transactions on Visualization and Computer Graphics.

[13]  Andrew Zisserman,et al.  Video Google: a text retrieval approach to object matching in videos , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[14]  B. S. Manjunath,et al.  Introduction to MPEG-7: Multimedia Content Description Interface , 2002 .

[15]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Marc Alexa,et al.  How do humans sketch objects? , 2012, ACM Trans. Graph..

[17]  Andrew Zisserman,et al.  Learning an Alphabet of Shape and Appearance for Multi-Class Object Detection , 2008, International Journal of Computer Vision.

[18]  Marc Alexa,et al.  An evaluation of descriptors for large-scale image retrieval from sketched feature lines , 2010, Comput. Graph..

[19]  Rui Hu,et al.  Gradient field descriptor for sketch based retrieval and localization , 2010, 2010 IEEE International Conference on Image Processing.

[20]  Yang Song,et al.  Learning Fine-Grained Image Similarity with Deep Ranking , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Eli Shechtman,et al.  Matching Local Self-Similarities across Images and Videos , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Abdolah Chalechale,et al.  Sketch-based image matching Using Angular partitioning , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[23]  Vipin Kumar,et al.  Face Recognition using Line Edge Map , 2014 .

[24]  Jon Louis Bentley,et al.  Multidimensional binary search trees used for associative searching , 1975, CACM.

[25]  H. Knutsson.  Representing Local Structure Using Tensors , 1989 .

[26]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[27]  Rui Hu,et al.  A bag-of-regions approach to sketch-based image retrieval , 2011, 2011 18th IEEE International Conference on Image Processing.

[28]  Tao Xiang,et al.  Deep Multi-task Attribute-driven Ranking for Fine-grained Sketch-based Image Retrieval , 2016, BMVC.

[29]  Toshikazu Kato,et al.  A sketch retrieval method for full color image database-query by visual example , 1992, [1992] Proceedings. 11th IAPR International Conference on Pattern Recognition.

[30]  J. V. Shinde,et al.  Survey on Sketch Based Image Retrieval , 2016 .

[31]  Qian Yu,et al.  Sketch-aNet that Beats Humans , 2015 .

[32]  S. B. Bagal,et al.  Survey paper on Sketch Based and Content Based Image Retrieval , 2015 .

[33]  James Ze Wang,et al.  Image retrieval: Ideas, influences, and trends of the new age , 2008, CSUR.

[34]  Laurent Wendling,et al.  Symbol recognition using spatial relations , 2012, Pattern Recognit. Lett..

[35]  JUSTIN ZOBEL,et al.  Inverted files for text search engines , 2006, CSUR.

[36]  Shih-Fu Chang,et al.  Image Retrieval: Current Techniques, Promising Directions, and Open Issues , 1999, J. Vis. Commun. Image Represent..

[37]  Alberto Del Bimbo,et al.  Visual Image Retrieval by Elastic Matching of User Sketches , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[38]  Shih-Fu Chang,et al.  Image retrieval with sketches and compositions , 2000, 2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No.00TH8532).

[39]  Liqing Zhang,et al.  Edgel index for large-scale sketch-based image search , 2011, CVPR 2011.

[40]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[41]  George R. Thoma,et al.  RSILC: Rotation- and Scale-Invariant, Line-based Color-aware descriptor , 2015, Image Vis. Comput..

[42]  Tao Xiang,et al.  Sketch-a-Net that Beats Humans , 2015, BMVC.

[43]  Cordelia Schmid,et al.  Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[44]  David G. Lowe,et al.  Fast Approximate Nearest Neighbors with Automatic Algorithm Configuration , 2009, VISAPP.

[45]  Shaogang Gong,et al.  Free-Hand Sketch Synthesis with Deformable Stroke Models , 2016, International Journal of Computer Vision.

[46]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[47]  Ghazali Sulong,et al.  A SKETCH BASED IMAGE RETRIEVAL: A REVIEW OF LITERATURE , 2014 .

[48]  David A. McAllester,et al.  Object Detection with Discriminatively Trained Part Based Models , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[49]  Rui Hu,et al.  A performance evaluation of gradient field HOG descriptor for sketch based image retrieval , 2013, Comput. Vis. Image Underst..

[50]  M. Kendall.  A NEW MEASURE OF RANK CORRELATION , 1938 .

[51]  Carl-Fredrik Westin,et al.  Representing Local Structure Using Tensors II , 2011, SCIA.

[52]  Alberto Del Bimbo,et al.  Visual image retrieval by elastic deformation of object sketches , 1994, Proceedings of 1994 IEEE Symposium on Visual Languages.

[53]  Shaogang Gong,et al.  Intra-category sketch-based image retrieval by matching deformable part models , 2014, BMVC.

[54]  Thorsten Joachims,et al.  Optimizing search engines using clickthrough data , 2002, KDD.

[55]  Laurent Wendling,et al.  Dtw-Radon-Based Shape Descriptor for Pattern Recognition , 2013, Int. J. Pattern Recognit. Artif. Intell..

[56]  Jianguo Zhang,et al.  The PASCAL Visual Object Classes Challenge , 2006 .

[57]  Toshikazu Kato,et al.  Query by Visual Example - Content based Image Retrieval , 1992, EDBT.

[58]  Daniel P. Lopresti,et al.  Feature-based approach for image retrieval by sketch , 1997, Other Conferences.

[59]  Xiaogang Wang,et al.  Face sketch recognition , 2004, IEEE Transactions on Circuits and Systems for Video Technology.

[60]  James Hays,et al.  The sketchy database , 2016, ACM Trans. Graph..

[61]  Feng Liu,et al.  Sketch Me That Shoe , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[62]  Christos Faloutsos,et al.  QBIC project: querying images by content, using color, texture, and shape , 1993, Electronic Imaging.

[63]  Yann LeCun,et al.  Learning a similarity metric discriminatively, with application to face verification , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).