Painting style transfer for head portraits using convolutional neural networks

Head portraits are popular in traditional painting. Automating portrait painting is challenging as the human visual system is sensitive to the slightest irregularities in human faces. Applying generic painting techniques often deforms facial structures. On the other hand portrait painting techniques are mainly designed for the graphite style and/or are based on image analogies; an example painting as well as its original unpainted version are required. This limits their domain of applicability. We present a new technique for transferring the painting from a head portrait onto another. Unlike previous work our technique only requires the example painting and is not restricted to a specific style. We impose novel spatial constraints by locally transferring the color distributions of the example painting. This better captures the painting texture and maintains the integrity of facial structures. We generate a solution through Convolutional Neural Networks and we present an extension to video. Here motion is exploited in a way to reduce temporal inconsistencies and the shower-door effect. Our approach transfers the painting style while maintaining the input photograph identity. In addition it significantly reduces facial deformations over state of the art.

[1]  Leon A. Gatys,et al.  A Neural Algorithm of Artistic Style , 2015, ArXiv.

[2]  Andrew Hunter,et al.  Learnable Stroke Models for Example-based Portrait Painting , 2013, BMVC.

[3]  Michael Ashikhmin,et al.  Fast Texture Transfer , 2003, IEEE Computer Graphics and Applications.

[4]  Harry Shum,et al.  PicToon: a personalized image-based cartoon system , 2002, MULTIMEDIA '02.

[5]  Edward H. Adelson,et al.  Compressing and companding high dynamic range images with subband architectures , 2005, ACM Trans. Graph..

[6]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[7]  Frédo Durand,et al.  Two-scale tone management for photographic look , 2006, ACM Trans. Graph..

[8]  Jia-Guang Sun,et al.  Efficient example-based painting and synthesis of 2D directional texture , 2004, IEEE Transactions on Visualization and Computer Graphics.

[9]  Kyunghyun Yoon,et al.  Directional texture transfer , 2010, NPAR.

[10]  Michael Ashikhmin,et al.  Synthesizing natural textures , 2001, I3D '01.

[11]  Pawan Sinha,et al.  Face Recognition by Humans: Nineteen Results All Computer Vision Researchers Should Know About , 2006, Proceedings of the IEEE.

[12]  Meng Meng.  Artistic PaperCut of Human Portraits , 2013 .

[13]  Dani Lischinski,et al.  A Closed-Form Solution to Natural Image Matting , 2008 .

[14]  Andrea Vedaldi,et al.  Understanding deep image representations by inverting them , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[16]  Frédo Durand,et al.  De-emphasis of distracting image regions using texture power maps , 2005, APGV '05.

[17]  David Salesin,et al.  Image Analogies , 2001, SIGGRAPH.

[18]  Ming Yang,et al.  DeepFace: Closing the Gap to Human-Level Performance in Face Verification , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Song-Chun Zhu,et al.  Artistic paper-cut of human portraits , 2010, ACM Multimedia.

[20]  Irfan A. Essa,et al.  Image and video based painterly animation , 2004, NPAR '04.

[21]  LischinskiDani,et al.  A Closed-Form Solution to Natural Image Matting , 2008 .

[22]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[23]  Paul Haeberli,et al.  Paint by numbers: abstract image representations , 1990, SIGGRAPH.

[24]  Steve DiPaola,et al.  Painterly rendered portraits from photographs using a knowledge-based approach , 2007, Electronic Imaging.

[25]  Xiaogang Wang,et al.  Face Photo-Sketch Synthesis and Recognition , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Edward H. Adelson,et al.  Compressing and companding high dynamic range images with subband architectures , 2005, SIGGRAPH 2005.

[27]  Erik Reinhard,et al.  Human facial illustrations: Creation and psychophysical evaluation , 2004, TOGS.

[28]  Song-Chun Zhu,et al.  Portrait painting using active templates , 2011, NPAR '11.

[29]  Aaron Hertzmann,et al.  Painterly rendering with curved brush strokes of multiple sizes , 1998, SIGGRAPH.

[30]  John Collomosse,et al.  Painterly rendering using image salience , 2002, Proceedings 20th Eurographics UK Conference.

[31]  John P. Collomosse,et al.  Genetic Paint: A Search for Salient Paintings , 2005, EvoWorkshops.

[32]  N. Kanwisher,et al.  Can generic expertise explain special processing for faces? , 2007, Trends in Cognitive Sciences.

[33]  David Salesin,et al.  Image-Based Hair Capture by Inverse Lighting , 2002, Graphics Interface.

[34]  David S. Ebert,et al.  Stippling by example , 2009, NPAR '09.

[35]  Timothy F. Cootes,et al.  Active Shape Models-Their Training and Application , 1995, Comput. Vis. Image Underst..

[36]  Liang Lin,et al.  Painterly animation using video semantics and feature correspondence , 2010, NPAR.

[37]  Alexei A. Efros,et al.  Texture synthesis by non-parametric sampling , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[38]  A.C. Kokaram,et al.  N-dimensional probability density function transfer and its application to color transfer , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[39]  Thaddeus Beier,et al.  Feature-based image metamorphosis , 1992, SIGGRAPH.

[40]  Barbara J. Meier.  Painterly rendering for animation , 1996, SIGGRAPH.

[41]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[42]  Aaron Hertzmann,et al.  Paint by relaxation , 2001, Proceedings. Computer Graphics International 2001.

[43]  Aaron Hertzmann,et al.  AniPaint: Interactive Painterly Animation from Video , 2012, IEEE Transactions on Visualization and Computer Graphics.

[44]  Frédo Durand,et al.  Style transfer for headshot portraits , 2014, ACM Trans. Graph..

[45]  Nanning Zheng,et al.  Example-based facial sketch generation with non-parametric sampling , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[46]  Tinghuai Wang,et al.  Video stylization for digital ambient displays of home movies , 2010, NPAR.

[47]  Tomoyuki Nishita,et al.  Creating various styles of animations using example-based filtering , 2003, Proceedings Computer Graphics International 2003.

[48]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[49]  Hong Chen,et al.  Example-based Automatic Portraiture , 2002 .

[50]  Tobias Isenberg,et al.  State of the "Art”: A Taxonomy of Artistic Stylization Techniques for Images and Video , 2013, IEEE Transactions on Visualization and Computer Graphics.

[51]  Amit R.Sharma,et al.  Face Photo-Sketch Synthesis and Recognition , 2012 .

[52]  Alexei A. Efros,et al.  Image quilting for texture synthesis and transfer , 2001, SIGGRAPH.

[53]  Peter Litwinowicz,et al.  Processing images and video for an impressionist effect , 1997, SIGGRAPH.

[54]  Simon Lucey,et al.  Face alignment through subspace constrained mean-shifts , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[55]  R. Poeckert,et al.  Warping digital images using thin plate splines , 1993, Pattern Recognit..

[56]  Adam Finkelstein,et al.  Interactive painterly stylization of images, videos and 3D animations , 2010, I3D '10.

[57]  Antonio Torralba,et al.  SIFT Flow: Dense Correspondence across Different Scenes , 2008, ECCV.

[58]  Peter Shirley,et al.  Artistic Vision: painterly rendering using computer vision techniques , 2002, NPAR '02.

[59]  Song-Chun Zhu,et al.  From image parsing to painterly rendering , 2009, TOGS.