Chinese flower-bird character generation based on pencil drawings or brush drawings

Abstract. Chinese flower-bird characters are gems of traditional Chinese art. It is an artistic font and the strokes of the characters are designed as beautiful patterns of flowers, birds, etc. The generation of such characters requires painters’ great efforts. Imagine that if we only need to sketch the outline of the ideal flower-bird characters using a pencil, and then we can quickly obtain these artistic characters, which will be of great significance in promoting their development, allowing more people to appreciate and even create this art by computer. Recently, with the development of deep learning and the invention of generative adversarial networks (GANs), some studies on font generation have made new progress. However, there is no research on the generation of flower-bird characters. We provide a solution by designing a GAN-based architecture to generate flower-bird characters. More specifically, a generator inspired by U-Net translates pencil drawings or brush drawings to flower-bird characters, and a patch-level discriminator distinguishes whether the received image is real. In addition to adversarial loss, a valid loss term called structural similarity loss is designed to further drive the network to generate satisfactory images. The quantitative analysis and user perceptual validation show the effectiveness of our method.

[1]  Cewu Lu,et al.  Combining sketch and tone for pencil drawing production , 2012, NPAR '12.

[2]  Andrea Vedaldi,et al.  Improved Texture Networks: Maximizing Quality and Diversity in Feed-Forward Stylization and Texture Synthesis , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Serge J. Belongie,et al.  Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[4]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Jie Li,et al.  Font generation based on least squares conditional generative adversarial nets , 2017, Multimedia Tools and Applications.

[6]  Zunlei Feng,et al.  Neural Style Transfer: A Review , 2017, IEEE Transactions on Visualization and Computer Graphics.

[7]  Chuan Li,et al.  Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks , 2016, ECCV.

[8]  Timo Aila,et al.  A Style-Based Generator Architecture for Generative Adversarial Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Yi-Hsuan Yang,et al.  MidiNet: A Convolutional Generative Adversarial Network for Symbolic-Domain Music Generation , 2017, ISMIR.

[10]  Thomas S. Huang,et al.  Generative Image Inpainting with Contextual Attention , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[11]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[12]  Yuke Zhu,et al.  StrokeBank: Automating Personalized Chinese Handwriting Generation , 2014, AAAI.

[13]  Connelly Barnes,et al.  Stable and Controllable Neural Texture Synthesis and Style Transfer Using Histogram Losses , 2017, ArXiv.

[14]  Jonathon Shlens,et al.  Conditional Image Synthesis with Auxiliary Classifier GANs , 2016, ICML.

[15]  Jan Kautz,et al.  Learning a manifold of fonts , 2014, ACM Trans. Graph..

[16]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[17]  Wenyu Liu,et al.  Auto-Encoder Guided GAN for Chinese Calligraphy Synthesis , 2017, 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR).

[18]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[19]  Hang Zhang,et al.  Multi-style Generative Network for Real-time Transfer , 2017, ECCV Workshops.

[20]  Yunhe Pan,et al.  Automatic generation of artistic chinese calligraphy , 2004, IEEE Intelligent Systems.

[21]  Hyunsoo Kim,et al.  Learning to Discover Cross-Domain Relations with Generative Adversarial Networks , 2017, ICML.

[22]  Bernhard Schölkopf,et al.  EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[23]  Leon A. Gatys,et al.  Image Style Transfer Using Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[25]  Antoni B. Chan,et al.  FlexyFont: Learning Transferring Rules for Flexible Typeface Synthesis , 2015, Comput. Graph. Forum.

[26]  Taesung Park,et al.  Semantic Image Synthesis With Spatially-Adaptive Normalization , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Wei Xiong,et al.  Learning to Generate Time-Lapse Videos Using Multi-stage Dynamic Generative Adversarial Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[28]  Jan Kautz,et al.  High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[29]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[30]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[32]  David Zhang,et al.  FSIM: A Feature Similarity Index for Image Quality Assessment , 2011, IEEE Transactions on Image Processing.

[33]  Masaki Nakagawa,et al.  Generating realistic Kanji character images from on-line patterns , 2001, Proceedings of Sixth International Conference on Document Analysis and Recognition.

[34]  Camille Couprie,et al.  Semantic Segmentation using Adversarial Networks , 2016, NIPS 2016.

[35]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[36]  Keiji Yanai,et al.  Neural Font Style Transfer , 2017, 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR).

[37]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[38]  Tat-Seng Chua,et al.  Laplacian-Steered Neural Style Transfer , 2017, ACM Multimedia.

[39]  Kaiqi Huang,et al.  GP-GAN: Towards Realistic High-Resolution Image Blending , 2017, ACM Multimedia.

[40]  Hao Jiang,et al.  Automatic Generation of Personal Chinese Handwriting by Capturing the Characteristics of Personal Handwriting , 2009, IAAI.

[41]  Hiroshi Ishikawa,et al.  Globally and locally consistent image completion , 2017, ACM Trans. Graph..

[42]  Fernando Argelaguet Sanz,et al.  Automatic speed graph generation for predefined camera paths , 2009, SG 2009.

[43]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Jeff Donahue,et al.  Large Scale GAN Training for High Fidelity Natural Image Synthesis , 2018, ICLR.

[45]  Hao Jiang,et al.  Automatic Generation of Chinese Calligraphic Writings with Style Imitation , 2009, IEEE Intelligent Systems.

[46]  Chuan Li,et al.  Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[47]  Takeo Igarashi,et al.  Example-Based Automatic Font Generation , 2010, Smart Graphics.

[48]  Lei Zhang,et al.  Gradient Magnitude Similarity Deviation: A Highly Efficient Perceptual Image Quality Index , 2013, IEEE Transactions on Image Processing.

[49]  Bo Zhao,et al.  Automatic generation of large-scale handwriting fonts via style learning , 2016, SIGGRAPH Asia Technical Briefs.

[50]  Yunchao Wei,et al.  Perceptual Generative Adversarial Networks for Small Object Detection , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Jan Kautz,et al.  Unsupervised Image-to-Image Translation Networks , 2017, NIPS.

[52]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[53]  Hao Li,et al.  High-Resolution Image Inpainting Using Multi-scale Neural Patch Synthesis , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[54]  Jan Kautz,et al.  Multimodal Unsupervised Image-to-Image Translation , 2018, ECCV.