Visio-lization: generating novel facial images

Our goal is to generate novel realistic images of faces using a model trained from real examples. This model consists of two components: First we consider face images as samples from a texture with spatially varying statistics and describe this texture with a local non-parametric model. Second, we learn a parametric global model of all of the pixel values. To generate realistic faces, we combine the strengths of both approaches and condition the local non-parametric model on the global parametric model. We demonstrate that with appropriate choice of local and global models it is possible to reliably generate new realistic face images that do not correspond to any individual in the training data. We extend the model to cope with considerable intra-class variation (pose and illumination). Finally, we apply our model to editing real facial images: we demonstrate image in-painting, interactive techniques for improving synthesized images and modifying facial expressions.

[1]  Geoffrey E. Hinton,et al.  The EM algorithm for mixtures of factor analyzers , 1996 .

[2]  Christoph Bregler,et al.  Video Rewrite: Driving Visual Speech with Audio , 1997, SIGGRAPH.

[3]  Matthew Turk,et al.  A Morphable Model For The Synthesis Of 3D Faces , 1999, SIGGRAPH.

[4]  Alexei A. Efros,et al.  Texture synthesis by non-parametric sampling , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[5]  Joshua B. Tenenbaum,et al.  Separating Style and Content with Bilinear Models , 2000, Neural Computation.

[6]  Marc Levoy,et al.  Fast texture synthesis using tree-structured vector quantization , 2000, SIGGRAPH.

[7]  Alexei A. Efros,et al.  Image quilting for texture synthesis and transfer , 2001, SIGGRAPH.

[8]  Michael Ashikhmin,et al.  Synthesizing natural textures , 2001, I3D '01.

[9]  Irfan A. Essa,et al.  Graphcut textures: image and video synthesis using graph cuts , 2003, ACM Trans. Graph..

[10]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[11]  Ramesh C. Jain,et al.  Content Based Image Synthesis , 2004, CIVR.

[12]  John Hart,et al.  ACM Transactions on Graphics , 2004, SIGGRAPH 2004.

[13]  David Salesin,et al.  Interactive digital photomontage , 2004, ACM Trans. Graph..

[14]  Tony Ezzat,et al.  Trainable videorealistic speech animation , 2002, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[15]  T. Kanade,et al.  High-zoom video hallucination by exploiting spatio-temporal regularities , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[16]  Hans-Peter Seidel,et al.  Creating Face Models from Vague Mental Images , 2005, SIGGRAPH '05.

[17]  Dahua Lin,et al.  Hallucinating faces: TensorPatch super-resolution and coupled residue compensation , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[18]  Harry Shum,et al.  Face Hallucination: Theory and Practice , 2007, International Journal of Computer Vision.

[19]  M. Gross,et al.  Analysis of human faces using a measurement-based skin reflectance model , 2006, ACM Trans. Graph..

[20]  Radford M. Neal Pattern Recognition and Machine Learning , 2007, Technometrics.

[21]  Alexei A. Efros,et al.  Scene completion using millions of photographs , 2007, SIGGRAPH 2007.

[22]  Alexei A. Efros,et al.  Photo clip art , 2007, ACM Trans. Graph..

[23]  Jonathan Warrell,et al.  Tied Factor Analysis for Face Recognition across Large Pose Differences , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Shree K. Nayar,et al.  Face swapping: automatically replacing faces in photographs , 2008, SIGGRAPH 2008.

[25]  Alexei A. Efros,et al.  Image‐based Shaving , 2008, Comput. Graph. Forum.

[26]  V. Kshirsagar,et al.  Face recognition using Eigenfaces , 2011, 2011 3rd International Conference on Computer Research and Development.