Recent Progress of Face Image Synthesis

Face synthesis has been a fascinating yet challenging problem in computer vision and machine learning. Its main research effort is to design algorithms to generate photo-realistic face images via given semantic domain. It has been a crucial prepossessing step of main-stream face recognition approaches and an excellent test of AI ability to use complicated probability distributions. In this paper, we provide a comprehensive review of typical face synthesis works that involve traditional methods as well as advanced deep learning approaches. Particularly, Generative Adversarial Net (GAN) is highlighted to generate photo-realistic and identity preserving results. Furthermore, the public available databases and evaluation metrics are introduced in details. We end the review with discussing unsolved difficulties and promising directions for future research.

[1]  Xiaogang Wang,et al.  Deep Learning Identity-Preserving Face Space , 2013, 2013 IEEE International Conference on Computer Vision.

[2]  Ming-Yu Liu,et al.  Coupled Generative Adversarial Networks , 2016, NIPS.

[3]  Takeo Kanade,et al.  Multi-PIE , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[4]  Scott E. Reed,et al.  Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis , 2015, NIPS.

[5]  Dominic M. Dwyer,et al.  The utility of multiple synthesized views in the recognition of unfamiliar faces , 2016, Quarterly journal of experimental psychology.

[6]  Xuelong Li,et al.  Multiple Representations-Based Face Sketch–Photo Synthesis , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[7]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[8]  Xinyu Huang,et al.  Mask-off: Synthesizing Face Images in the Presence of Head-mounted Displays , 2016, 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[9]  Bo Zhao,et al.  Multi-View Image Generation from a Single-View , 2017, ACM Multimedia.

[10]  Hongyuan Zha,et al.  Manifold Based Face Synthesis from Sparse Samples , 2013, 2013 IEEE International Conference on Computer Vision.

[11]  Jon Gauthier.  Conditional generative adversarial nets for convolutional face generation , 2015 .

[12]  Patrick J. Flynn,et al.  SREFI: Synthesis of realistic example face images , 2017, 2017 IEEE International Joint Conference on Biometrics (IJCB).

[13]  Marwan Mattar,et al.  Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments , 2008 .

[14]  Dimitris N. Metaxas,et al.  Reconstruction for Feature Disentanglement in Pose-invariant Face Recognition , 2017, ArXiv.

[15]  Stefanos Zafeiriou,et al.  Robust Statistical Face Frontalization , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[16]  Liang Chang,et al.  Face Sketch Synthesis via Sparse Representation , 2010, 2010 20th International Conference on Pattern Recognition.

[17]  Jie Li,et al.  Face sketch synthesis via sparse representation-based greedy search. , 2015, IEEE transactions on image processing : a publication of the IEEE Signal Processing Society.

[18]  Chi-Keung Tang,et al.  Conditional CycleGAN for Attribute Guided Face Image Generation , 2017, ArXiv.

[19]  Xiangyang Xue,et al.  Semi-Latent GAN: Learning to generate and modify facial images from attributes , 2017, ArXiv.

[20]  Tien D. Bui,et al.  Deep Appearance Models: A Deep Boltzmann Machine Approach for Face Modeling , 2016, International Journal of Computer Vision.

[21]  Ming-Hsuan Yang,et al.  Generative Face Completion , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Dhruv Batra,et al.  LR-GAN: Layered Recursive Generative Adversarial Networks for Image Generation , 2016, ICLR.

[23]  Justus Thies,et al.  InverseFaceNet: Deep Monocular Inverse Face Rendering , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[24]  Yi Yang,et al.  GeneGAN: Learning Object Transfiguration and Attribute Subspace from Unpaired Data , 2017, BMVC 2017.

[25]  Mahantapas Kundu,et al.  Face Synthesis (FASY) System for Generation of a Face Image from Human Description , 2010, ArXiv.

[26]  Xinbo Gao,et al.  Random sampling for fast face sketch synthesis , 2017, Pattern Recognit..

[27]  Ruigang Yang,et al.  Manifold Estimation in View-Based Feature Space for Face Synthesis across Poses , 2009, ACCV.

[28]  Mark Sandler,et al.  Inverting face embeddings with convolutional neural networks , 2016, ArXiv.

[29]  Dimitris N. Metaxas,et al.  Reconstruction-Based Disentanglement for Pose-Invariant Face Recognition , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[30]  Andrew Zisserman,et al.  Deep Face Recognition , 2015, BMVC.

[31]  Chris Donahue,et al.  Semantically Decomposing the Latent Spaces of Generative Adversarial Networks , 2017, ICLR.

[32]  Volker Blanz,et al.  Automated 3D Face Reconstruction from Multiple Images Using Quality Measures , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Alexei A. Efros,et al.  Image‐based Shaving , 2008, Comput. Graph. Forum.

[34]  Michael Elad,et al.  Example-Based Image Synthesis via Randomized Patch-Matching , 2016, IEEE Transactions on Image Processing.

[35]  Huijun Gao,et al.  Structured detail enhancement for cross-modality face synthesis , 2016, Neurocomputing.

[36]  Feng Li,et al.  Detail-Enhanced Cross-Modality Face Synthesis via Guided Image Filtering , 2015, CCCV.

[37]  Matan Sela,et al.  Learning Detailed Face Reconstruction from a Single Image , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Yen-Wei Chen,et al.  Tensor-based subspace learning and its applications in multi-pose face synthesis , 2010, Neurocomputing.

[39]  Zhenan Sun,et al.  DeMeshNet: Blind Face Inpainting for Deep MeshFace Verification , 2016, IEEE Transactions on Information Forensics and Security.

[40]  Justus Thies,et al.  InverseFaceNet: Deep Single-Shot Inverse Face Rendering From A Single Image , 2017, ArXiv.

[41]  Gang Hua,et al.  Face Relighting from a Single Image under Arbitrary Unknown Lighting Conditions , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Chi-Keung Tang,et al.  Attribute-Guided Face Generation Using Conditional CycleGAN , 2017, ECCV.

[43]  Feng Liu,et al.  On 3D face reconstruction via cascaded regression in shape space , 2015, Frontiers of Information Technology & Electronic Engineering.

[44]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[45]  Yunhong Wang,et al.  Face synthesis from low-resolution near-infrared to high-resolution visual light spectrum based on tensor analysis , 2014, Neurocomputing.

[46]  Léon Bottou,et al.  Wasserstein Generative Adversarial Networks , 2017, ICML.

[47]  Xiaoming Liu,et al.  Disentangled Representation Learning GAN for Pose-Invariant Face Recognition , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Feng Liu,et al.  Joint Face Alignment and 3D Face Reconstruction , 2016, ECCV.

[49]  Yueting Zhuang,et al.  Expressive 3D face synthesis by multi-space modeling , 2008, 2008 First IEEE International Conference on Ubi-Media Computing.

[50]  Amit R.Sharma,et al.  Face Photo-Sketch Synthesis and Recognition , 2012 .

[51]  Yann LeCun,et al.  Energy-based Generative Adversarial Network , 2016, ICLR.

[52]  Hyunsoo Kim,et al.  Learning to Discover Cross-Domain Relations with Generative Adversarial Networks , 2017, ICML.

[53]  Xiaoming Liu,et al.  Coefficients Pose-Variant Input Recogni 8 on Engine Frontalized Output Generator FF-GAN D Discriminator Extreme Pose Input Frontalized Output , 2017 .

[54]  Matan Sela,et al.  3D Face Reconstruction by Learning from Synthetic Data , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[55]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[56]  William T. Freeman,et al.  Face Synthesis from Facial Identity Features , 2017, ArXiv.

[57]  Xiaogang Wang,et al.  Multi-View Perceptron: a Deep Model for Learning Face Identity and View Representations , 2014, NIPS.

[58]  Zhenan Sun,et al.  Multi-task ConvNet for blind face inpainting with application to face verification , 2016, 2016 International Conference on Biometrics (ICB).

[59]  Yiying Tong,et al.  FaceWarehouse: A 3D Facial Expression Database for Visual Computing , 2014, IEEE Transactions on Visualization and Computer Graphics.

[60]  Marios Savvides,et al.  Facial Ethnic Appearance Synthesis , 2014, ECCV Workshops.

[61]  Ran He,et al.  Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesis , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[62]  William A. P. Smith,et al.  Driving 3D morphable models using shading cues , 2012, Pattern Recognit..

[63]  Hui Zhang,et al.  A framework of face synthesis based on multilinear analysis , 2016, VRCAI.

[64]  Taehyun Rhee,et al.  A probabilistic approach to realistic face synthesis , 2010, 2010 IEEE International Conference on Image Processing.

[65]  Chunna Tian,et al.  Face Sketch Synthesis using E-HMM and Selective Ensemble , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[66]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[67]  Jean-Luc Dugelay,et al.  Face aging with conditional generative adversarial networks , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[68]  Marios Savvides,et al.  Gender and Ethnicity Specific Generic Elastic Models from a Single 2D Image for Novel 2D Pose Face Synthesis and Recognition , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[69]  Yann LeCun,et al.  Energy-based Generative Adversarial Networks , 2016, ICLR.

[70]  Alberto Del Bimbo,et al.  Effective 3D based frontalization for unconstrained face recognition , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[71]  Mahantapas Kundu,et al.  Construction of human faces from textual descriptions , 2011, Soft Comput..

[72]  Erik Learned-Miller,et al.  Labeled Faces in the Wild : Updates and New Reporting Procedures , 2014 .

[73]  Xinbo Gao,et al.  Sparse frontal face image synthesis from an arbitrary profile image , 2014, Neurocomputing.

[74]  William T. Freeman,et al.  Synthesizing Normalized Faces from Facial Identity Features , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[75]  Xinbo Gao,et al.  Random Sampling and Locality Constraint for Face Sketch , 2017, ArXiv.

[76]  Ioannis A. Kakadiaris,et al.  Robust 3D Face Shape Reconstruction from Single Images via Two-Fold Coupled Structure Learning and Off-the-Shelf Landmark Detectors , 2014, BMVC.

[77]  Xinbo Gao,et al.  Face Sketch Synthesis via Sparse Representation-Based Greedy Search , 2015, IEEE Transactions on Image Processing.

[78]  Soodamani Ramalingam,et al.  3D Face Synthesis with KINECT , 2013, 2013 IEEE International Conference on Systems, Man, and Cybernetics.

[79]  Marios Savvides,et al.  3-D Generic Elastic Models for Fast and Texture Preserving 2-D Novel Pose Synthesis , 2012, IEEE Transactions on Information Forensics and Security.

[80]  Shengcai Liao,et al.  Learning Face Representation from Scratch , 2014, ArXiv.

[81]  Ming Yang,et al.  DeepFace: Closing the Gap to Human-Level Performance in Face Verification , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[82]  Wojciech Zaremba,et al.  Improved Techniques for Training GANs , 2016, NIPS.

[83]  Sina Honari,et al.  Learning to Generate Samples from Noise through Infusion Training , 2017, ICLR.

[84]  Léon Bottou,et al.  Wasserstein GAN , 2017, ArXiv.