Query-by-sketch image retrieval using homogeneous painting style characterization

Abstract. As an effective and efficient way to graphically portray ideas and concepts, free-hand sketches are playing an increasingly significant role in today’s image retrieval systems. There are many factors that contribute to a successful sketch-based image retrieval (SBIR) system, but perhaps the most essential ones are the design of powerful image feature descriptors and effective perceptual similarity metrics in the feature space. However, the painting experiences and styles of users vary hugely, which poses a great challenge for consistently accurate feature representation and matching. We introduce an approach for SBIR by characterizing homogeneous sketch drawing styles. Specifically, given a sketch query, the proposed method first infers the user’s painting style from it by analyzing contour features. Then, top K-nearest sketches of historical users with similar drawing characters are identified from the database. Finally, the matched images of the query can be obtained by reranking those of the K-nearest sketches stored in the historical recordings. To evaluate the effectiveness of the proposed method, we conducted quantitative comparisons with several peer SBIR methods. Experimental results indicate that the proposed method is 15% higher than state-of-the-art in terms of mean average precision.

[1]  Jitendra Malik,et al.  Shape matching and object recognition using shape contexts , 2010, 2010 3rd International Conference on Computer Science and Information Technology.

[2]  J. Bobadilla,et al.  Recommender systems survey , 2013, Knowl. Based Syst..

[3]  David Heckerman,et al.  Empirical Analysis of Predictive Algorithms for Collaborative Filtering , 1998, UAI.

[4]  Jose M. Saavedra,et al.  Sketch based image retrieval using a soft computation of the histogram of edge local orientations (S-HELO) , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[5]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[6]  Rui Hu,et al.  A bag-of-regions approach to sketch-based image retrieval , 2011, 2011 18th IEEE International Conference on Image Processing.

[7]  John P. Collomosse,et al.  Scalable Sketch-Based Image Retrieval Using Color Gradient Features , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[8]  Yehuda Koren,et al.  Scalable Collaborative Filtering with Jointly Derived Neighborhood Interpolation Weights , 2007, Seventh IEEE International Conference on Data Mining (ICDM 2007).

[9]  Gediminas Adomavicius,et al.  Personalization technologies: A process-oriented perspective , 2006, Wirtschaftsinf..

[10]  Andrew Blake,et al.  "GrabCut" , 2004, ACM Trans. Graph..

[11]  Ivan Ganchev,et al.  A trust-enriched approach for item-based collaborative filtering recommendations , 2016, 2016 IEEE 12th International Conference on Intelligent Computer Communication and Processing (ICCP).

[12]  Haibin Ling,et al.  Shape Classification Using the Inner-Distance , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  K. P. Soman,et al.  Vector Space Model as Cognitive Space for Text Classification , 2017, ArXiv.

[14]  Rui Hu,et al.  A performance evaluation of gradient field HOG descriptor for sketch based image retrieval , 2013, Comput. Vis. Image Underst..

[15]  Richi Nayak,et al.  Do-Rank: DCG Optimization for Learning-to-Rank in Tag-Based Item Recommendation Systems , 2015, PAKDD.

[16]  Lior Rokach,et al.  Introduction to Recommender Systems Handbook , 2011, Recommender Systems Handbook.

[17]  George Karypis,et al.  Item-based top-N recommendation algorithms , 2004, TOIS.

[18]  Ling Shao,et al.  Deep Sketch Hashing: Fast Free-Hand Sketch-Based Image Retrieval , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Hasan Dag,et al.  Improving item-based recommendation accuracy with user's preferences on Apache Mahout , 2016, 2016 IEEE International Conference on Big Data (Big Data).

[20]  Pasquale Lops,et al.  Content-based Recommender Systems: State of the Art and Trends , 2011, Recommender Systems Handbook.

[21]  Fernando Ortega,et al.  A non negative matrix factorization for collaborative filtering recommender systems based on a Bayesian probabilistic model , 2016, Knowl. Based Syst..

[22]  Fei Wang,et al.  A Data‐Driven Approach for Sketch‐Based 3D Shape Retrieval via Similar Drawing‐Style Recommendation , 2017, Comput. Graph. Forum.

[23]  Franca Garzotto,et al.  Content-Based Video Recommendation System Based on Stylistic Visual Features , 2016, Journal on Data Semantics.

[24]  Junhao Wen,et al.  SVM-TIA a shilling attack detection method based on SVM and target item analysis in recommender systems , 2016, Neurocomputing.

[25]  Ruslan Salakhutdinov,et al.  Probabilistic Matrix Factorization , 2007, NIPS.

[26]  Zhengtao Yu,et al.  A social recommendation method based on trust propagation and singular value decomposition , 2017, J. Intell. Fuzzy Syst..

[27]  Ryutarou Ohbuchi,et al.  Visual Saliency Weighting and Cross-Domain Manifold Ranking for Sketch-Based Image Retrieval , 2014, MMM.

[28]  Yehuda Koren,et al.  Advances in Collaborative Filtering , 2011, Recommender Systems Handbook.

[29]  Dunja Mladenic,et al.  kNN Versus SVM in the Collaborative Filtering Framework , 2006, Data Science and Classification.

[30]  Benjamin Bustos,et al.  An Improved Histogram of Edge Local Orientations for Sketch-Based Image Retrieval , 2010, DAGM-Symposium.

[31]  Marc Alexa,et al.  An evaluation of descriptors for large-scale image retrieval from sketched feature lines , 2010, Comput. Graph..

[32]  Patrick Seemann,et al.  Matrix Factorization Techniques for Recommender Systems , 2014 .

[33]  Anurag Mittal,et al.  Similarity-Invariant Sketch-Based Image Retrieval in Large Databases , 2014, ECCV.

[34]  Greg Linden,et al.  Amazon . com Recommendations Item-to-Item Collaborative Filtering , 2001 .

[35]  Toshikazu Kato,et al.  Query by Visual Example - Content based Image Retrieval , 1992, EDBT.

[36]  Makbule Gulcin Ozsoy,et al.  From Word Embeddings to Item Recommendation , 2016, ArXiv.

[37]  Thomas Hofmann,et al.  Collaborative filtering via gaussian probabilistic latent semantic analysis , 2003, SIGIR.

[38]  Alexander Felfernig,et al.  Toward the Next Generation of Recommender Systems: Applications and Research Challenges , 2013 .

[39]  Yuting Zhang,et al.  Sketch-Based Image Retrieval by Salient Contour Reinforcement , 2016, IEEE Transactions on Multimedia.

[40]  Yong Cheng,et al.  Deep Multimodal Embedding Model for Fine-grained Sketch-based Image Retrieval , 2017, SIGIR.

[41]  John Riedl,et al.  Analysis of recommendation algorithms for e-commerce , 2000, EC '00.

[42]  Domonkos Tikk,et al.  Matrix factorization and neighbor based algorithms for the netflix prize problem , 2008, RecSys '08.

[43]  Barry Smyth,et al.  User-Based Opinion-based Recommendation , 2017, IJCAI.

[44]  Liqing Zhang,et al.  Sketch-based image retrieval on a large scale database , 2012, ACM Multimedia.

[45]  George Karypis,et al.  A Comprehensive Survey of Neighborhood-based Recommendation Methods , 2011, Recommender Systems Handbook.

[46]  John Riedl,et al.  GroupLens: an open architecture for collaborative filtering of netnews , 1994, CSCW '94.

[47]  Shu Wang,et al.  Sketch-Based Image Retrieval Through Hypothesis-Driven Object Boundary Selection With HLR Descriptor , 2015, IEEE Transactions on Multimedia.

[48]  Bela Gipp,et al.  Research-paper recommender systems: a literature survey , 2015, International Journal on Digital Libraries.

[49]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[50]  Ming-Hsuan Yang,et al.  Sketch retrieval via local dense stroke features , 2016, Image Vis. Comput..

[51]  Marc Alexa,et al.  Sketch-Based Image Retrieval: Benchmark and Bag-of-Features Descriptors , 2011, IEEE Transactions on Visualization and Computer Graphics.

[52]  Jurij F. Tasic,et al.  Affective Labeling in a Content-Based Recommender System for Images , 2013, IEEE Transactions on Multimedia.

[53]  Andrew Zisserman,et al.  Efficient retrieval of deformable shape classes using local self-similarities , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[54]  Rui Hu,et al.  Gradient field descriptor for sketch based retrieval and localization , 2010, 2010 IEEE International Conference on Image Processing.

[55]  Pasquale Lops,et al.  Semantics-aware Content-based Recommender Systems , 2014, CBRecSys@RecSys.

[56]  Devavrat Shah,et al.  Regret Guarantees for Item-Item Collaborative Filtering , 2015 .

[57]  Shaogang Gong,et al.  Intra-category sketch-based image retrieval by matching deformable part models , 2014, BMVC.