Perceptual-Aware Sketch Simplification Based on Integrated VGG Layers

Deep learning has been recently demonstrated as an effective tool for raster-based sketch simplification. Nevertheless, it remains challenging to simplify extremely rough sketches. We found that a simplification network trained with a simple loss, such as pixel loss or discriminator loss, may fail to retain the semantically meaningful details when simplifying a very sketchy and complicated drawing. In this paper, we show that, with a well-designed multi-layer perceptual loss, we are able to obtain aesthetic and neat simplification results preserving semantically important global structures as well as fine details without blurriness and excessive emphasis on local structures. To do so, we design a multi-layer discriminator by fusing all VGG feature layers to differentiate sketches and clean lines. The weights used in layer fusing are automatically learned via an intelligent adjustment mechanism. Furthermore, to evaluate our method, we compare our method to state-of-the-art methods through multiple experiments, including visual comparison and intensive user study.

[1]  Adrien Bousseau,et al.  Fidelity vs. simplicity , 2016, ACM Trans. Graph..

[2]  Markus H. Gross,et al.  Topology-driven vectorization of clean line drawings , 2013, ACM Trans. Graph..

[3]  James Hays,et al.  The sketchy database , 2016, ACM Trans. Graph..

[4]  Tao Xiang,et al.  Making better use of edges via perceptual grouping , 2015, CVPR.

[5]  K. Sasaki,et al.  Learning to simplify , 2016, ACM Trans. Graph..

[6]  Pingkun Yan,et al.  CT Image Denoising with Perceptive Deep Neural Networks , 2017, ArXiv.

[7]  Karl Tombre,et al.  Robust and accurate vectorization of line drawings , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[9]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Baoquan Chen,et al.  Efficient and Dynamic Simplification of Line Drawings , 2008, Comput. Graph. Forum.

[11]  Jitendra Malik,et al.  Hypercolumns for object segmentation and fine-grained localization , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Hiroshi Ishikawa,et al.  Mastering Sketching: Adversarial Augmentation for Structured Prediction , 2017 .

[13]  Fisher Yu,et al.  Scribbler: Controlling Deep Image Synthesis with Sketch and Color , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Ravin Balakrishnan,et al.  ILoveSketch: as-natural-as-possible sketching system for creating 3d curve models , 2008, UIST '08.

[15]  Minh N. Do,et al.  Semantic Image Inpainting with Deep Generative Models , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Tien-Tsin Wong,et al.  Closure-aware sketch simplification , 2015, ACM Trans. Graph..

[17]  Kwan-Liu Ma,et al.  Rendering complexity in computer-generated pen-and-ink illustrations , 2004, NPAR '04.

[18]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Hao Li,et al.  High-Resolution Image Inpainting Using Multi-scale Neural Patch Synthesis , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  James Arvo,et al.  Fluid sketches: continuous recognition and morphing of simple hand-drawn shapes , 2000, UIST '00.

[22]  Alexei A. Efros,et al.  Colorful Image Colorization , 2016, ECCV.

[23]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[24]  Bernhard Preim,et al.  Tuning rendered line-drawings , 1995 .

[25]  Huchuan Lu,et al.  Amulet: Aggregating Multi-level Convolutional Features for Salient Object Detection , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[26]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[27]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[28]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[29]  Gregory Shakhnarovich,et al.  Learning Representations for Automatic Colorization , 2016, ECCV.