Compositional Model-Based Sketch Generator in Facial Entertainment

Face sketch synthesis (FSS) plays an important role in facial entertainment, which includes face sketch morphing among two styles, multiview FSS and face sketch expression manipulation. For facial entertainment, most existing FSS methods generate sketches with over-smoothing effects, i.e., fine details are suppressed more or less. In this paper, we propose a face sketch generator based on the compositional model to handle this issue. It decomposes a face into different components instead of patches as before, and each component has several candidate templates. Multilevel B-spline approximation is utilized to delicately polish the chosen templates of all components. To fuse these components, Poisson blending is employed instead of the weighted average operator. The proposed compositional method crucially reduces the high frequency loss and improves the synthesis performance in comparison to the state-of-the-art methods. Experiments on face sketch morphing, expression manipulation, and multiview FSS, make further efforts to demonstrate the effectiveness of the proposed method.

[1]  Jiri Matas,et al.  XM2VTSDB: The Extended M2VTS Database , 1999 .

[2]  Fernando De la Torre,et al.  Supervised Descent Method and Its Applications to Face Alignment , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[3]  Xuelong Li,et al.  Local face sketch synthesis learning , 2008, Neurocomputing.

[4]  Harry Shum,et al.  To appear in the ACM SIGGRAPH conference proceedings Drag-and-Drop Pasting , 2022 .

[5]  Aleix M. Martinez,et al.  The AR face database , 1998 .

[6]  Xinbo Gao,et al.  Photo-sketch synthesis and recognition based on subspace learning , 2010, Neurocomputing.

[7]  Qi Tian,et al.  Multiview Face Recognition: From TensorFace to V-TensorFace and K-TensorFace , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[8]  Adil Masood Siddiqui,et al.  A locally constrained radial basis function for registration and warping of images , 2009, Pattern Recognit. Lett..

[9]  Xuelong Li,et al.  Heterogeneous image transformation , 2013, Pattern Recognit. Lett..

[10]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[11]  Xintao Hu,et al.  Weakly supervised target detection in remote sensing images based on transferred deep features and negative bootstrapping , 2016, Multidimens. Syst. Signal Process..

[12]  Xuelong Li,et al.  A new approach for face recognition by sketches in photos , 2009, Signal Process..

[13]  Dani Lischinski,et al.  A Closed-Form Solution to Natural Image Matting , 2008 .

[14]  Fei Gao,et al.  Biologically inspired image quality assessment , 2016, Signal Process..

[15]  Ming-Hsuan Yang,et al.  Real-Time Exemplar-Based Face Sketch Synthesis , 2014, ECCV.

[16]  Jie Li,et al.  Bayesian Face Sketch Synthesis , 2017, IEEE Transactions on Image Processing.

[17]  Xuelong Li,et al.  Color to Gray: Visual Cue Preservation , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Xuelong Li,et al.  Universal Blind Image Quality Assessment Metrics Via Natural Scene Statistics and Multiple Kernel Learning , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[19]  Xuelong Li,et al.  Transductive Face Sketch-Photo Synthesis , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[20]  Heng Yang,et al.  Facial feature point detection: A comprehensive survey , 2014, Neurocomputing.

[21]  Xuelong Li,et al.  Face Sketch–Photo Synthesis and Retrieval Using Sparse Representation , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[22]  Xiaogang Wang,et al.  Lighting and Pose Robust Face Sketch Synthesis , 2010, ECCV.

[23]  Xinbo Gao,et al.  Random Sampling and Locality Constraint for Face Sketch , 2017, ArXiv.

[24]  Aljoscha Smolic,et al.  Automatic View Synthesis by Image-Domain-Warping , 2013, IEEE Transactions on Image Processing.

[25]  Zeev Farbman,et al.  Coordinates for instant image cloning , 2009, ACM Trans. Graph..

[26]  Chunna Tian,et al.  Face Sketch Synthesis using E-HMM and Selective Ensemble , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[27]  David J. Kriegman,et al.  Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection , 1996, ECCV.

[28]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[29]  Michael F. Cohen,et al.  Simultaneous Matting and Compositing , 2006, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[30]  Xiaogang Wang,et al.  Face sketch synthesis and recognition , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[31]  Hao Zhou,et al.  Markov Weight Fields for face sketch synthesis , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[32]  Anton Schwaighofer,et al.  Transductive and Inductive Methods for Approximate Gaussian Process Regression , 2002, NIPS.

[33]  Sung Yong Shin,et al.  Image Metamorphosis with Scattered Feature Constraints , 1996, IEEE Trans. Vis. Comput. Graph..

[34]  Xuelong Li,et al.  Learning to Rank for Blind Image Quality Assessment , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[35]  Hanqing Lu,et al.  A nonlinear approach for face sketch synthesis and recognition , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[36]  Wei Liu,et al.  Bayesian Tensor Inference for Sketch-Based Facial Photo Hallucination , 2007, IJCAI.

[37]  Timothy F. Cootes,et al.  Active Shape Models-Their Training and Application , 1995, Comput. Vis. Image Underst..

[38]  Sung Yong Shin,et al.  Scattered Data Interpolation with Multilevel B-Splines , 1997, IEEE Trans. Vis. Comput. Graph..

[39]  Bin Song,et al.  Unified framework for face sketch synthesis , 2017, Signal Process..

[40]  Fernando Pérez-Cruz,et al.  Empirical risk minimization for support vector classifiers , 2003, IEEE Trans. Neural Networks.

[41]  Thaddeus Beier,et al.  Feature-based image metamorphosis , 1998 .

[42]  Xinbo Gao,et al.  Recognition of facial sketch styles , 2015, Neurocomputing.

[43]  Xiaogang Wang,et al.  Face photo recognition using sketch , 2002, Proceedings. International Conference on Image Processing.

[44]  Amit R.Sharma,et al.  Face Photo-Sketch Synthesis and Recognition , 2012 .

[45]  Dani Lischinski,et al.  Coordinates for instant image cloning , 2009, SIGGRAPH 2009.

[46]  Sung Yong Shin,et al.  Image Morphing Using Deformation Techniques , 1996, Comput. Animat. Virtual Worlds.

[47]  Betty Edwards,et al.  The new drawing on the right side of the brain workbook : guided practice in the five basic skills of drawing , 1979 .

[48]  제라드 메디오니,et al.  3d face reconstruction from 2d images , 2007 .

[49]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[50]  Xinbo Gao,et al.  Robust Face Sketch Style Synthesis , 2016, IEEE Transactions on Image Processing.

[51]  Luc Van Gool,et al.  SURF: Speeded Up Robust Features , 2006, ECCV.

[52]  Leon A. Gatys,et al.  A Neural Algorithm of Artistic Style , 2015, ArXiv.