Line Artist: A Multiple Style Sketch to Painting Synthesis Scheme

Drawing a beautiful painting is a dream of many people since childhood. In this paper, we propose a novel scheme, Line Artist, to synthesize artistic style paintings with freehand sketch images, leveraging the power of deep learning and advanced algorithms. Our scheme includes three models. The Sketch Image Extraction (SIE) model is applied to generate the training data. It includes smoothing reality images and pencil sketch extraction. The Detailed Image Synthesis (DIS) model trains a conditional generative adversarial network to generate detailed real-world information. The Adaptively Weighted Artistic Style Transfer (AWAST) model is capable to combine multiple style images with a content with the VGG19 network and PageRank algorithm. The appealing artistic images are then generated by optimization iterations. Experiments are operated on the Kaggle Cats dataset and The Oxford Buildings Dataset. Our synthesis results are proved to be artistic, beautiful and robust.

[1]  Michael Elad,et al.  Style Transfer Via Texture Synthesis , 2016, IEEE Transactions on Image Processing.

[2]  Andrea Vedaldi,et al.  Texture Networks: Feed-forward Synthesis of Textures and Stylized Images , 2016, ICML.

[3]  Jonathon Shlens,et al.  Conditional Image Synthesis with Auxiliary Classifier GANs , 2016, ICML.

[4]  Lei Yang,et al.  An improved Sobel edge detection , 2010, 2010 3rd International Conference on Computer Science and Information Technology.

[5]  Xinbo Gao,et al.  Face Sketch Synthesis via Sparse Representation-Based Greedy Search , 2015, IEEE Transactions on Image Processing.

[6]  Vladlen Koltun,et al.  Photographic Image Synthesis with Cascaded Refinement Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[7]  Fay Huang,et al.  Synthesis of Oil-Style Paintings , 2015, PSIVT.

[8]  Cewu Lu,et al.  Image smoothing via L0 gradient minimization , 2011, ACM Trans. Graph..

[9]  Rajeev Motwani,et al.  The PageRank Citation Ranking : Bringing Order to the Web , 1999, WWW 1999.

[10]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Oscar Castillo,et al.  An improved sobel edge detection method based on generalized type-2 fuzzy logic , 2014, Soft Computing.

[12]  Leon A. Gatys,et al.  A Neural Algorithm of Artistic Style , 2015, ArXiv.

[13]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Rada Mihalcea,et al.  TextRank: Bringing Order into Text , 2004, EMNLP.

[15]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[16]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[17]  Min H. Kim,et al.  Laplacian Patch-Based Image Synthesis , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Fisher Yu,et al.  Scribbler: Controlling Deep Image Synthesis with Sketch and Color , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Vince Grolmusz,et al.  A note on the PageRank of undirected graphs , 2012, Inf. Process. Lett..

[21]  Shiming Xiang,et al.  Segment Graph Based Image Filtering: Fast Structure-Preserving Smoothing , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[22]  Chuan Li,et al.  Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Cewu Lu,et al.  Combining sketch and tone for pencil drawing production , 2012, NPAR '12.

[24]  Zulin Wang,et al.  Road Structure Refined CNN for Road Extraction in Aerial Image , 2017, IEEE Geoscience and Remote Sensing Letters.

[25]  C. Lawrence Zitnick,et al.  Structured Forests for Fast Edge Detection , 2013, 2013 IEEE International Conference on Computer Vision.