Semi-supervised reference-based sketch extraction using a contrastive learning framework

Sketches reflect the drawing style of individual artists; therefore, it is important to consider their unique styles when extracting sketches from color images for various applications. Unfortunately, most existing sketch extraction methods are designed to extract sketches of a single style. Although there have been some attempts to generate various style sketches, the methods generally suffer from two limitations: low quality results and difficulty in training the model due to the requirement of a paired dataset. In this paper, we propose a novel multi-modal sketch extraction method that can imitate the style of a given reference sketch with unpaired data training in a semi-supervised manner. Our method outperforms state-of-the-art sketch extraction methods and unpaired image translation methods in both quantitative and qualitative evaluations.

[1]  Chang Wook Seo,et al.  Reference Based Sketch Extraction via Attention Mechanism , 2022, ACM Trans. Graph..

[2]  Seoung Wug Oh,et al.  StylePortraitVideo: Editing Portrait Videos with Expression Optimization , 2022, Comput. Graph. Forum.

[3]  A. Savchenko HSEmotion: High-speed emotion recognition library , 2022, Softw. Impacts.

[4]  R. Stiefelhagen,et al.  Pose-based Contrastive Learning for Domain Agnostic Activity Representations , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[5]  S. Lim,et al.  Neural Image Recolorization for Creative Domains , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[6]  F. Durand,et al.  Learning to generate line drawings that convey geometry and semantics , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Amit H. Bermano,et al.  CLIPasso , 2022, ACM Trans. Graph..

[8]  D. Griol,et al.  A Proposal for Multimodal Emotion Recognition Using Aural Transformers and Action Units on RAVDESS Dataset , 2021, Applied Sciences.

[9]  Xueting Liu,et al.  Reference-guided structure-aware deep sketch colorization for cartoons , 2021, Computational Visual Media.

[10]  Mingming Gong,et al.  Unaligned Image-to-Image Translation by Learning to Reweight , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[11]  Pinaki Nath Chowdhury,et al.  SketchLattice: Latticed Representation for Sketch Manipulation , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[12]  E. Simo-Serra,et al.  General virtual sketching framework for vector line art , 2021, ACM Trans. Graph..

[13]  Edgar Simo-Serra,et al.  Line Art Colorization with Concatenated Spatial Attention , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[14]  Xueting Liu,et al.  Deep Style Transfer for Line Drawings , 2021, AAAI.

[15]  Tien-Tsin Wong,et al.  Perceptual-Aware Sketch Simplification Based on Integrated VGG Layers , 2021, IEEE Transactions on Visualization and Computer Graphics.

[16]  Bernt Schiele,et al.  You Only Need Adversarial Supervision for Semantic Image Synthesis , 2020, ICLR.

[17]  Eli Shechtman,et al.  Spatially-Adaptive Pixelwise Networks for Fast Image Translation , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  David Vanderhaeghe,et al.  A benchmark for rough sketch cleanup , 2020, ACM Trans. Graph..

[19]  Thomas Lukasiewicz,et al.  Lightweight Generative Adversarial Networks for Text-Guided Image Manipulation , 2020, NeurIPS.

[20]  Nicu Sebe,et al.  Describe What to Change: A Text-guided Unsupervised Image-to-image Translation Approach , 2020, ACM Multimedia.

[21]  Alexei A. Efros,et al.  Contrastive Learning for Unpaired Image-to-Image Translation , 2020, ECCV.

[22]  Alexei A. Efros,et al.  Swapping Autoencoder for Deep Image Manipulation , 2020, NeurIPS.

[23]  Paul L. Rosin,et al.  Unpaired Portrait Drawing Generation via Asymmetric Cycle Mapping , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Jaegul Choo,et al.  Reference-Based Sketch Image Colorization Using Augmented-Self Reference and Dense Semantic Correspondence , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Luc Van Gool,et al.  SESAME: Semantic Editing of Scenes by Adding, Manipulating or Erasing Objects , 2020, ECCV.

[26]  Limin Wang,et al.  SketchyCOCO: Image Generation From Freehand Scene Sketches , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Wen Gao,et al.  Direct Speech-to-Image Translation , 2020, IEEE Journal of Selected Topics in Signal Processing.

[28]  Bingchen Liu,et al.  Sketch-to-Art: Synthesizing Stylized Art Images From Sketches , 2020, ACCV.

[29]  Geoffrey E. Hinton,et al.  A Simple Framework for Contrastive Learning of Visual Representations , 2020, ICML.

[30]  Philip H. S. Torr,et al.  Multi-Channel Attention Selection GANs for Guided Image-to-Image Translation , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Thomas Lukasiewicz,et al.  ManiGAN: Text-Guided Image Manipulation , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Jung-Woo Ha,et al.  StarGAN v2: Diverse Image Synthesis for Multiple Domains , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  A. Tal,et al.  Breaking the Cycle – Colleagues Are All You Need , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Chengying Gao,et al.  Language-based colorization of scene sketches , 2019, ACM Trans. Graph..

[35]  Xuejin Chen,et al.  LinesToFacePhoto: Face Photo Generation From Lines With Conditional Self-Attention Generative Adversarial Networks , 2019, ACM Multimedia.

[36]  Mehran Ebrahimi,et al.  Artist-Guided Semiautomatic Animation Colorization , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[37]  Eunhyeok Park,et al.  Tag2Pix: Line Art Colorization Using Text Tag With SECat and Changing Loss , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[38]  Ran Yi,et al.  APDrawingGAN: Generating Artistic Portrait Drawings From Face Photos With Hierarchical GANs , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Maneesh Kumar Singh,et al.  DRIT++: Diverse Image-to-Image Translation via Disentangled Representations , 2019, International Journal of Computer Vision.

[40]  Nicu Sebe,et al.  Multi-Channel Attention Selection GAN With Cascaded Semantic Guidance for Cross-View Image Translation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Haifeng Hu,et al.  Multimodal supervised image translation , 2019, Electronics Letters.

[42]  Tien-Tsin Wong,et al.  Two-stage sketch colorization , 2018, ACM Trans. Graph..

[43]  Haojie Li,et al.  User-Guided Deep Anime Line Art Colorization with Conditional Adversarial Networks , 2018, ACM Multimedia.

[44]  In-So Kweon,et al.  CBAM: Convolutional Block Attention Module , 2018, ECCV.

[45]  Dongdong Chen,et al.  Deep exemplar-based colorization , 2018, ACM Trans. Graph..

[46]  Wei Wang,et al.  Multistage Adversarial Losses for Pose-Based Human Image Synthesis , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[47]  T. Tuytelaars,et al.  Exemplar Guided Unsupervised Image-to-Image Translation , 2018, ICLR.

[48]  Jan Kautz,et al.  Multimodal Unsupervised Image-to-Image Translation , 2018, ECCV.

[49]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[50]  James Hays,et al.  SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[51]  Nicu Sebe,et al.  Deformable GANs for Pose-Based Human Image Generation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[52]  Jan Kautz,et al.  High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[53]  Chao Wang,et al.  Discriminative Region Proposal Adversarial Networks for High-Quality Image-to-Image Translation , 2017, ECCV.

[54]  Xueting Liu,et al.  Deep extraction of manga structural lines , 2017, ACM Trans. Graph..

[55]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[56]  Alexei A. Efros,et al.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[57]  Hiroshi Ishikawa,et al.  Mastering Sketching , 2017, ACM Trans. Graph..

[58]  Serge J. Belongie,et al.  Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[59]  Tomasz Malisiewicz,et al.  RoomNet: End-to-End Room Layout Estimation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[60]  Hyunsoo Kim,et al.  Learning to Discover Cross-Domain Relations with Generative Adversarial Networks , 2017, ICML.

[61]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[62]  Björn Stenger,et al.  Pano2CAD: Room Layout from a Single Panorama Image , 2016, 2017 IEEE Winter Conference on Applications of Computer Vision (WACV).

[63]  K. Sasaki,et al.  Learning to simplify , 2016, ACM Trans. Graph..

[64]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[65]  Timothy M. Hospedales,et al.  Free-Hand Sketch Synthesis with Deformable Stroke Models , 2015, International Journal of Computer Vision.

[66]  Saining Xie,et al.  Holistically-Nested Edge Detection , 2015, International Journal of Computer Vision.

[67]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[68]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[69]  Holger Winnemöller,et al.  XDoG: advanced image stylization with eXtended Difference-of-Gaussians , 2011, NPAR '11.

[70]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[71]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[72]  S. P. Lloyd,et al.  Least squares quantization in PCM , 1982, IEEE Trans. Inf. Theory.

[73]  Jong Chul Ye,et al.  DiffusionCLIP: Text-guided Image Manipulation Using Diffusion Models , 2021, ArXiv.

[74]  Jie Zhou,et al.  Structural Deep Metric Learning for Room Layout Estimation , 2020, ECCV.

[75]  Cordelia Schmid,et al.  IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2004, Washington, DC, USA, June 27 - July 2, 2004 , 2004, CVPR Workshops.