Context-Aware Asset Search for Graphic Design

Graphic design tools provide powerful controls for expert-level design creation, but the options can often be overwhelming for novices. This paper proposes Context-Aware Asset Search tools that take the current state of the user's design into account, thereby providing search and selections that are compatible with the current design and better fit the user's needs. In particular, we focus on image search and color selection, two tasks that are central to design. We learn a model for compatibility of images and colors within a design, using crowdsourced data. We then use the learned model to rank image search results or color suggestions during design. We found counterintuitive behavior using conventional training with pairwise comparisons for image search, where models with and without compatibility performed similarly. We describe a data collection procedure that alleviates this problem. We show that our method outperforms baseline approaches in quantitative evaluation, and we also evaluate a prototype interactive design tool.

[1]  Tovi Grossman,et al.  Design and evaluation of a command recommendation system for software applications , 2011, TCHI.

[2]  Filip Radlinski,et al.  A support vector method for optimizing average precision , 2007, SIGIR.

[3]  Ehud Rivlin,et al.  Placing search in context: the concept revisited , 2002, TOIS.

[4]  Robinson Piramuthu,et al.  Large scale visual recommendations from street fashion images , 2014, KDD.

[5]  Hailin Jin,et al.  Composition-Preserving Deep Photo Aesthetics Assessment , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  R. A. Bradley,et al.  Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons , 1952 .

[7]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[8]  Trevor Darrell,et al.  Recognizing Image Style , 2013, BMVC.

[9]  Aaron Hertzmann,et al.  Exploratory font selection using crowdsourced attributes , 2014, ACM Trans. Graph..

[10]  Frédo Durand,et al.  Learning Visual Importance for Graphic Designs and Data Visualizations , 2017, UIST.

[11]  Falk Scholer,et al.  The Benefits of Magnitude Estimation Relevance Assessments for Information Retrieval Evaluation , 2015, SIGIR.

[12]  Rynson W. H. Lau,et al.  Directing user attention via visual flow on web designs , 2016, ACM Trans. Graph..

[13]  Stephen DiVerdi,et al.  Palette-based photo recoloring , 2015, ACM Trans. Graph..

[14]  Kristen Grauman,et al.  Relative attributes , 2011, 2011 International Conference on Computer Vision.

[15]  Wilmot Li,et al.  Style compatibility for 3D furniture models , 2015, ACM Trans. Graph..

[16]  Hinrich Schütze,et al.  Personalized search , 2002, CACM.

[17]  Jaana Kekäläinen,et al.  IR evaluation methods for retrieving highly relevant documents , 2000, SIGIR '00.

[18]  Pat Hanrahan,et al.  Modeling how people extract color themes from images , 2013, CHI.

[19]  Changsheng Xu,et al.  Hi, magic closet, tell me what to wear! , 2012, ACM Multimedia.

[20]  Jan P. Allebach,et al.  Colors -- Messengers of Concepts: Visual Design Mining for Learning Color Semantics , 2015, ArXiv.

[21]  Pat Hanrahan,et al.  Probabilistic color-by-numbers , 2013, ACM Trans. Graph..

[22]  Ron Kohavi,et al.  Responsible editor: R. Bayardo. , 2022 .

[23]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[24]  Levent Burak Kara,et al.  Semantic shape editing using deformation handles , 2015, ACM Trans. Graph..

[25]  Vittorio Ferrari,et al.  End-to-End Training of Object Class Detectors for Mean Average Precision , 2016, ACCV.

[26]  Scott R. Klemmer,et al.  Example-centric programming: integrating web search into the development environment , 2010, CHI.

[27]  Siddhartha Chaudhuri,et al.  Attribit: content creation with semantic attributes , 2013, UIST.

[28]  Pere Obrador.  Automatic color scheme picker for document templates based on image analysis and dual problem , 2006, Electronic Imaging.

[29]  Thorsten Joachims,et al.  Optimizing search engines using clickthrough data , 2002, KDD.

[30]  Diego Gutierrez,et al.  A similarity measure for illustration style , 2014, ACM Trans. Graph..

[31]  Anton van den Hengel,et al.  Image-Based Recommendations on Styles and Substitutes , 2015, SIGIR.

[32]  Sean M. McNee,et al.  Being accurate is not enough: how accuracy metrics have hurt recommender systems , 2006, CHI Extended Abstracts.

[33]  Naila Murray,et al.  AVA: A large-scale database for aesthetic visual analysis , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[34]  Alla Sheffer,et al.  Elements of style , 2015, ACM Trans. Graph..

[35]  Aaron Hertzmann,et al.  Color compatibility from large datasets , 2011, SIGGRAPH 2011.

[36]  Serge J. Belongie,et al.  Learning Visual Clothing Style with Heterogeneous Dyadic Co-Occurrences , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[37]  David Maxwell Chickering,et al.  Here or there: preference judgments for relevance , 2008 .

[38]  Elisa Bertino,et al.  Quality Control in Crowdsourcing Systems: Issues and Directions , 2013, IEEE Internet Computing.

[39]  William A. Gale,et al.  A sequential algorithm for training text classifiers , 1994, SIGIR '94.

[40]  Kavita Bala,et al.  Learning visual similarity for product design with convolutional neural networks , 2015, ACM Trans. Graph..

[41]  Hailin Jin,et al.  Collaborative feature learning from social media , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Daniel Cohen-Or,et al.  Color harmonization , 2006, ACM Trans. Graph..

[44]  Takeo Igarashi,et al.  A suggestive interface for 3D drawing , 2001, SIGGRAPH Courses.