Enhancing the Realism of Sketch and Painted Portraits With Adaptable Patches

Realizing unrealistic faces is a complicated task that requires a rich imagination and comprehension of facial structures. When face matching, warping or stitching techniques are applied, existing methods are generally incapable of capturing detailed personal characteristics, are disturbed by block boundary artefacts, or require painting‐photo pairs for training. This paper presents a data‐driven framework to enhance the realism of sketch and portrait paintings based only on photo samples. It retrieves the optimal patches of adaptable shapes and numbers according to the content of the input portrait and collected photos. These patches are then seamlessly stitched by chromatic gain and offset compensation and multi‐level blending. Experiments and user evaluations show that the proposed method is able to generate realistic and novel results for a moderately sized photo collection.

[1]  C. Lawrence Zitnick,et al.  Binary Coherent Edge Descriptors , 2010, ECCV.

[2]  Alexei A. Efros,et al.  Data-driven visual similarity for cross-domain image matching , 2011, ACM Trans. Graph..

[3]  Bernard Tiddeman,et al.  Towards Realism in Facial Image Transformation: Results of a Wavelet MRF Method , 2005, Comput. Graph. Forum.

[4]  Song-Chun Zhu,et al.  Portrait painting using active templates , 2011, NPAR '11.

[5]  Anil K. Jain,et al.  Sketch-to-photo matching: a feature-based approach , 2010, Defense + Commercial Sensing.

[6]  Shree K. Nayar,et al.  Face swapping: automatically replacing faces in photographs , 2008, SIGGRAPH 2008.

[7]  Chun Chen,et al.  Face sketch-to-photo synthesis from simple line drawing , 2012, Proceedings of The 2012 Asia Pacific Signal and Information Processing Association Annual Summit and Conference.

[8]  Edward H. Adelson,et al.  A multiresolution spline with application to image mosaics , 1983, TOGS.

[9]  LiXuelong,et al.  Face sketch-photo synthesis under multi-dictionary sparse representation framework , 2011 .

[10]  Andrew Hunter,et al.  Learnable Stroke Models for Example-based Portrait Painting , 2013, BMVC.

[11]  Alexei A. Efros,et al.  Image‐based Shaving , 2008, Comput. Graph. Forum.

[12]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[13]  Andrew Zisserman,et al.  Automated mosaicing with super-resolution zoom , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[14]  A. Martínez,et al.  The AR face databasae , 1998 .

[15]  David Salesin,et al.  Image-Based Hair Capture by Inverse Lighting , 2002, Graphics Interface.

[16]  Po-Wen Cheng,et al.  SI-Cut: Structural Inconsistency Analysis for Image Foreground Extraction , 2015, IEEE Transactions on Visualization and Computer Graphics.

[17]  Vladimir Kolmogorov,et al.  An experimental comparison of min-cut/max- flow algorithms for energy minimization in vision , 2001, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  David Salesin,et al.  Synthesizing realistic facial expressions from photographs , 1998, SIGGRAPH.

[19]  Yunjin Lee,et al.  Feature‐guided Image Stippling , 2008, Comput. Graph. Forum.

[20]  David Salesin,et al.  Image Analogies , 2001, SIGGRAPH.

[21]  Aleix M. Martinez,et al.  The AR face database , 1998 .

[22]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[23]  Qinping Zhao,et al.  Artistic Illumination Transfer for Portraits , 2012, Comput. Graph. Forum.

[24]  Eli Shechtman,et al.  Brushables: Example‐based Edge‐aware Directional Texture Painting , 2015, Comput. Graph. Forum.

[25]  Fred Nicolls,et al.  Locating Facial Features with an Extended Active Shape Model , 2008, ECCV.

[26]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[27]  Xuelong Li,et al.  Face Sketch-Photo Synthesis under Multi-dictionary Sparse Representation Framework , 2011, 2011 Sixth International Conference on Image and Graphics.

[28]  Vladimir Kolmogorov,et al.  What energy functions can be minimized via graph cuts? , 2002, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  William T. Freeman,et al.  Learning Low-Level Vision , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[30]  Amit R.Sharma,et al.  Face Photo-Sketch Synthesis and Recognition , 2012 .

[31]  Alexei A. Efros,et al.  Image quilting for texture synthesis and transfer , 2001, SIGGRAPH.

[32]  Nadia Magnenat-Thalmann,et al.  Generating Animatable 3D Virtual Humans from Photographs , 2000, Comput. Graph. Forum.

[33]  Matthew A. Brown,et al.  Automatic Panoramic Image Stitching using Invariant Features , 2007, International Journal of Computer Vision.

[34]  Yu-Ting Tsai,et al.  Generating Pointillism Paintings Based on Seurat's Color Composition , 2013, Comput. Graph. Forum.

[35]  Jan Kautz,et al.  Visio-lization: generating novel facial images , 2009, ACM Trans. Graph..

[36]  Shiguang Shan,et al.  A Compositional and Dynamic Model for Face Aging , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Hans-Peter Seidel,et al.  Exchanging Faces in Images , 2004, Comput. Graph. Forum.

[38]  David Salesin,et al.  Interactive digital photomontage , 2004, ACM Trans. Graph..

[39]  Wojciech Matusik,et al.  CG2Real: Improving the Realism of Computer Generated Images Using a Large Collection of Photographs , 2011, IEEE Transactions on Visualization and Computer Graphics.