Interactive Sketch‐Driven Image Synthesis

We present an interactive system for composing realistic images of an object under arbitrary pose and appearance specified by sketching. Our system draws inspiration from a traditional illustration workflow: The user first sketches rough ‘masses’ of the object, as ellipses, to define an initial abstract pose that can then be refined with more detailed contours as desired. The system is made robust to partial or inaccurate sketches using a reduced‐dimensionality model of pose space learnt from a labelled collection of photos. Throughout the composition process, interactive visual feedback is provided to guide the user. Finally, the user's partial or complete sketch, complemented with appearance requirements, is used to constrain the automatic synthesis of a novel, high‐quality, realistic image.

[1]  Alexei A. Efros,et al.  Photo clip art , 2007, ACM Trans. Graph..

[2]  Neil D. Lawrence,et al.  Probabilistic Non-linear Principal Component Analysis with Gaussian Process Latent Variable Models , 2005, J. Mach. Learn. Res..

[3]  Tovi Grossman,et al.  Sketch-sketch revolution: an engaging tutorial system for guided sketching and application learning , 2011, UIST.

[4]  Jan Kautz,et al.  Visio-lization: generating novel facial images , 2009, ACM Trans. Graph..

[5]  Sanja Fidler,et al.  Detect What You Can: Detecting and Representing Objects Using Holistic Models and Body Parts , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Shi-Min Hu,et al.  PoseShop: Human Image Database Construction and Personalized Content Synthesis , 2013, IEEE Transactions on Visualization and Computer Graphics.

[7]  Shi-Min Hu,et al.  Sketch2Scene: sketch-based co-retrieval and co-placement of 3D models , 2013, ACM Trans. Graph..

[8]  Charles R. Giardina,et al.  Elliptic Fourier features of a closed contour , 1982, Comput. Graph. Image Process..

[9]  W. H. McMaster,et al.  Polarization and the Stokes Parameters , 1954 .

[10]  E. Grinspun,et al.  Synthesizing structured image hybrids , 2010, ACM Trans. Graph..

[11]  Shimon Ullman,et al.  Combining Top-Down and Bottom-Up Segmentation , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[12]  Jitendra Malik,et al.  Poselets: Body part detectors trained using 3D human pose annotations , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[13]  Eli Shechtman,et al.  Image melding , 2012, ACM Trans. Graph..

[14]  Roberto Cipolla,et al.  Semantic Photo Synthesis , 2006, Comput. Graph. Forum.

[15]  Iasonas Kokkinos,et al.  Understanding Objects in Detail with Fine-Grained Attributes , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Ian D. Reid,et al.  Nonlinear shape manifolds as shape priors in level set segmentation and tracking , 2011, CVPR 2011.

[17]  Daniel Dixon,et al.  iCanDraw: using sketch recognition and corrective feedback to assist a user in drawing human faces , 2010, CHI.

[18]  Adrien Bousseau,et al.  The drawing assistant: automated drawing guidance and feedback from photographs , 2013, UIST.

[19]  Hongxun Yao,et al.  DNN Flow: DNN Feature Pyramid based Image Matching , 2014, BMVC.

[20]  Koos Eissen,et al.  Sketching: Drawing Techniques for Product Designers , 2009 .

[21]  Jeff Sauro,et al.  The Factor Structure of the System Usability Scale , 2009, HCI.

[22]  Stephen Gould,et al.  PatchMatchGraph: Building a Graph of Dense Patch Correspondences for Label Transfer , 2012, ECCV.

[23]  Shi-Min Hu,et al.  Data‐Driven Object Manipulation in Images , 2012, Comput. Graph. Forum.

[24]  Robert C. Bolles,et al.  Parametric Correspondence and Chamfer Matching: Two New Techniques for Image Matching , 1977, IJCAI.

[25]  Yong Jae Lee,et al.  ShadowDraw: real-time user guidance for freehand drawing , 2011, SIGGRAPH 2011.

[26]  Marc Alexa,et al.  Photosketcher: Interactive Sketch-Based Image Synthesis , 2011, IEEE Computer Graphics and Applications.

[27]  Ralph R. Martin,et al.  PatchNet: a patch-based image representation for interactive library-driven image editing , 2013, ACM Trans. Graph..

[28]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[29]  Yong Jae Lee,et al.  ShadowDraw: real-time user guidance for freehand drawing , 2011, ACM Trans. Graph..

[30]  Andrew W. Fitzgibbon,et al.  The Joint Manifold Model for Semi-supervised Multi-valued Regression , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[31]  Shi-Min Hu,et al.  Sketch2Photo: internet image montage , 2009, ACM Trans. Graph..

[32]  David Salesin,et al.  Image Analogies , 2001, SIGGRAPH.