Deep texture cartoonization via unsupervised appearance regularization

Abstract Texture plays an important role in cartoon images to represent materials of objects and enrich visual attractiveness. However, manually crafting a cartoon texture is not easy, so amateurs usually directly use cartoon textures downloaded from the Internet. Unfortunately, Internet resources are quite limited and often patented, which restrict the users from generating visually pleasant and personalized cartoon textures. In this paper, we propose a deep learning based method to generate cartoon textures from natural textures. Different from the existing photo cartoonization methods that only aim to generate cartoonic images, the key to our method is to generate cartoon textures that are both cartoonic and regular. To achieve this goal, we propose a regularization module to generate a regular natural texture with similar appearance as the input, and a cartoonization module to cartoffonize the regularized natural texture into a regular cartoon texture. Our method successfully produces cartoonic and regular textures from various natural textures.

[1]  Holger Winnemöller,et al.  Real-time video abstraction , 2006, SIGGRAPH 2006.

[2]  XuYi,et al.  Image smoothing via L0 gradient minimization , 2011 .

[3]  A. Bouzerdoum,et al.  Texture Classification using Convolutional Neural Networks , 2006, TENCON 2006 - 2006 IEEE Region 10 Conference.

[4]  M. Kanatzidis,et al.  Ultralow thermal conductivity and high thermoelectric figure of merit in SnSe crystals , 2014, Nature.

[5]  Subhransu Maji,et al.  Deep filter banks for texture recognition and segmentation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Paul F. Whelan,et al.  Using filter banks in Convolutional Neural Networks for texture classification , 2016, Pattern Recognit. Lett..

[7]  Peng-Jen Chen,et al.  Endoscopic Submucosal Dissection for Early Colorectal Neoplasms: Clinical Experience in a Tertiary Medical Center in Taiwan , 2013, Gastroenterology research and practice.

[8]  Anil K. Jain,et al.  Texture Analysis , 2018, Handbook of Image Processing and Computer Vision.

[9]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[10]  Björn Ommer,et al.  Content and Style Disentanglement for Artistic Style Transfer , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[11]  Douglas DeCarlo,et al.  Stylization and abstraction of photographs , 2002, ACM Trans. Graph..

[12]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[13]  Luc Van Gool,et al.  The Synthesizability of Texture Examples , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Xavier Snelgrove,et al.  High-resolution multi-scale neural texture synthesis , 2017, SIGGRAPH Asia Technical Briefs.

[15]  Jan Kautz,et al.  Local Laplacian filters: edge-aware image processing with a Laplacian pyramid , 2011, SIGGRAPH 2011.

[16]  Jan Kautz,et al.  Local Laplacian filters , 2015, Commun. ACM.

[17]  Ping Li,et al.  Deep Texture Exemplar Extraction Based on Trimmed T-CNN , 2020, IEEE Transactions on Multimedia.

[18]  Wei Gao,et al.  Fast Video Multi-Style Transfer , 2020, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).

[19]  Robert M. Haralick,et al.  Textural Features for Image Classification , 1973, IEEE Trans. Syst. Man Cybern..

[20]  DeCarloDoug,et al.  Stylization and abstraction of photographs , 2002 .

[21]  Dani Lischinski,et al.  Non-stationary texture synthesis by adversarial expansion , 2018, ACM Trans. Graph..

[22]  G. Deng,et al.  An adaptive Gaussian filter for noise reduction and edge detection , 1993, 1993 IEEE Conference Record Nuclear Science Symposium and Medical Imaging Conference.

[23]  Li Xu,et al.  Structure extraction from texture via relative total variation , 2012, ACM Trans. Graph..

[24]  Yong-Jin Liu,et al.  CartoonGAN: Generative Adversarial Networks for Photo Cartoonization , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[25]  Yang Zhou,et al.  ETNet: Error Transition Network for Arbitrary Style Transfer , 2019, NeurIPS.

[26]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[27]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[28]  Jing Wu,et al.  Retinal Fundus Image Enhancement Using the Normalized Convolution and Noise Removing , 2016, Int. J. Biomed. Imaging.

[29]  Gajanand Gupta,et al.  Algorithm for Image Processing Using Improved Median Filter and Comparison of Mean, Median and Improved Median Filter , 2011 .

[30]  A. Frigo,et al.  Terlipressin given by continuous intravenous infusion versus intravenous boluses in the treatment of hepatorenal syndrome: A randomized controlled study , 2016, Hepatology.

[31]  M. Pietikäinen,et al.  TEXTURE ANALYSIS WITH LOCAL BINARY PATTERNS , 2004 .

[32]  Jinze Yu,et al.  Learning to Cartoonize Using White-Box Cartoon Representations , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  David Salesin,et al.  Image Analogies , 2001, SIGGRAPH.

[34]  Derek Nowrouzezahrai,et al.  Learning hatching for pen-and-ink illustration of surfaces , 2012, TOGS.

[35]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Seungyong Lee,et al.  Flow-Based Image Abstraction , 2009, IEEE Transactions on Visualization and Computer Graphics.

[37]  Chuan Li,et al.  Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks , 2016, ECCV.

[38]  Ming-Hsuan Yang,et al.  Collaborative Distillation for Ultra-Resolution Universal Style Transfer , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Rama Chellappa,et al.  Entropy rate superpixel segmentation , 2011, CVPR 2011.

[40]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Jian Sun,et al.  Guided Image Filtering , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Matti Pietikäinen,et al.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[43]  Jürgen Döllner,et al.  Image Abstraction by Structure Adaptive Filtering , 2008, TPCG.

[44]  Jan P. Allebach,et al.  Adaptive Bilateral Filter for Sharpness Enhancement and Noise Removal , 2007, 2007 IEEE International Conference on Image Processing.

[45]  Jan Kautz,et al.  Multimodal Unsupervised Image-to-Image Translation , 2018, ECCV.

[46]  Cewu Lu,et al.  Image smoothing via L0 gradient minimization , 2011, ACM Trans. Graph..

[47]  Leon A. Gatys,et al.  Image Style Transfer Using Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Maneesh Kumar Singh,et al.  DRIT++: Diverse Image-to-Image Translation via Disentangled Representations , 2019, International Journal of Computer Vision.

[49]  Andrea Vedaldi,et al.  Texture Networks: Feed-forward Synthesis of Textures and Stylized Images , 2016, ICML.

[50]  Jiaying Liu,et al.  Demystifying Neural Style Transfer , 2017, IJCAI.

[51]  Ran Yi,et al.  APDrawingGAN: Generating Artistic Portrait Drawings From Face Photos With Hierarchical GANs , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[52]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[53]  Xueting Li,et al.  Learning Linear Transformations for Fast Arbitrary Style Transfer , 2018, ArXiv.

[54]  Serge J. Belongie,et al.  Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[55]  Björn Ommer,et al.  A Style-Aware Content Loss for Real-time HD Style Transfer , 2018, ECCV.

[56]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[57]  Daniel P. Huttenlocher,et al.  Efficient Graph-Based Image Segmentation , 2004, International Journal of Computer Vision.