SketchPatch

The paradigm of image-to-image translation is leveraged for the benefit of sketch stylization via transfer of geometric textural details. Lacking the necessary volumes of data for standard training of translation systems, we advocate for operation at the patch level, where a handful of stylized sketches provide ample mining potential for patches featuring basic geometric primitives. Operating at the patch level necessitates special consideration of full sketch translation, as individual translation of patches with no regard to neighbors is likely to produce visible seams and artifacts at patch borders. Aligned pairs of styled and plain primitives are combined to form input hybrids containing styled elements around the border and plain elements within, and given as input to a seamless translation (ST) generator, whose output patches are expected to reconstruct the fully styled patch. An adversarial addition promotes generalization and robustness to diverse geometries at inference time, forming a simple and effective system for arbitrary sketch stylization, as demonstrated upon a variety of styles and sketches.

[1]  Alexei A. Efros,et al.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[2]  Serge J. Belongie,et al.  Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[3]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[4]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[5]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Eli Shechtman,et al.  Brushables: Example‐based Edge‐aware Directional Texture Painting , 2015, Comput. Graph. Forum.

[7]  Jürgen Branke,et al.  Survey: State of the Art , 2002 .

[8]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[9]  Leon A. Gatys,et al.  Image Style Transfer Using Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Radomír Mech,et al.  DecoBrush: drawing structured decorative patterns by example , 2014, ACM Trans. Graph..

[11]  Stephen DiVerdi,et al.  RealBrush: painting with examples of physical media , 2013, ACM Trans. Graph..

[12]  Marc Alexa,et al.  The Markov pen: online synthesis of free-hand drawing styles , 2015, NPAR '15.

[13]  Fang-Lue Zhang,et al.  A survey of the state-of-the-art in patch-based synthesis , 2017, Computational Visual Media.

[14]  Takeo Igarashi,et al.  Vignette: interactive texture design and manipulation with freeform gestures for pen-and-ink illustration , 2012, CHI.

[15]  Carolyn Kieran,et al.  Survey of the State of the Art , 2016 .

[16]  Mark W. Schmidt,et al.  Fast Patch-based Style Transfer of Arbitrary Style , 2016, ArXiv.

[17]  Trevor Darrell,et al.  Multi-content GAN for Few-Shot Font Style Transfer , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[18]  Ping Tan,et al.  DualGAN: Unsupervised Dual Learning for Image-to-Image Translation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[19]  Wenjing Wang,et al.  TET-GAN: Text Effects Transfer via Stylization and Destylization , 2019, AAAI.

[20]  Ming-Hsuan Yang,et al.  Universal Style Transfer via Feature Transforms , 2017, NIPS.

[21]  Hyunsoo Kim,et al.  Learning to Discover Cross-Domain Relations with Generative Adversarial Networks , 2017, ICML.

[22]  Eli Shechtman,et al.  Im2Pencil: Controllable Pencil Illustration From Photographs , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Sylvain Lefebvre,et al.  By–example synthesis of curvilinear structured patterns , 2013, Comput. Graph. Forum.

[24]  Christian Santoni,et al.  gTangle: a grammar for the procedural generation of tangle patterns , 2016, ACM Trans. Graph..

[25]  Gang Hua,et al.  Visual attribute transfer through deep image analogy , 2017, ACM Trans. Graph..

[26]  Jianguo Xiao,et al.  Artistic glyph image synthesis via one-stage few-shot learning , 2019, ACM Trans. Graph..

[27]  Eli Shechtman,et al.  Enhancing neural style transfer using patch-based synthesis , 2019, Expressive.

[28]  Jan Kautz,et al.  Multimodal Unsupervised Image-to-Image Translation , 2018, ECCV.

[29]  Antoni B. Chan,et al.  Patternista: learning element style compatibility and spatial composition for ring-based layout decoration , 2016, Expressive.

[30]  David Salesin,et al.  Image Analogies , 2001, SIGGRAPH.

[31]  Raymond Y. K. Lau,et al.  Least Squares Generative Adversarial Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[32]  Adam Finkelstein,et al.  The Generalized PatchMatch Correspondence Algorithm , 2010, ECCV.

[33]  Steve Marschner,et al.  Brush stroke synthesis with a generative adversarial network driven by physically based simulation , 2018, Expressive.