Replacement of Facial Parts in Images

It is interesting to edit facial appearance in images to create a desirable facial shape of persons. In this paper, we propose a novel method to modify facial appearance by replacing facial parts between arbitrarily paired images. To this end, our method consists of face segmentation, face reconstruction, mesh deformation and image editing. Given one source and one target image, the target image is first segmented into the front facial region and background image. Secondly, 3D facial models and relevant scene parameters are estimated from both images. Thirdly, the target facial part is replaced with the selected source part on the 3D mesh. Then, the new replaced 3D face is rendered into a facial image. Finally, the new facial image is generated by seamlessly blending the rendered image and background image. The main advantage of this method is that we transfer facial geometric information between images using 3D model, which can deal with arbitrarily paired images with the different facial viewpoint. We present several experimental results to show the effectiveness of our method and comparison with those existing methods to demonstrate that our method is more advantageous and flexible in terms of practical applications.

[1]  Anil K. Jain,et al.  Sketch based face recognition: Forensic vs. composite sketches , 2013, 2013 International Conference on Biometrics (ICB).

[2]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[3]  Tal Hassner,et al.  On Face Segmentation, Face Swapping, and Face Perception , 2017, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[4]  Shuang Chen,et al.  Face recognition based on subset selection via metric learning on manifold , 2015, Frontiers of Information Technology & Electronic Engineering.

[5]  Kun Zhou,et al.  Mesh editing with poisson-based gradient field manipulation , 2004, SIGGRAPH 2004.

[6]  Ruofeng Tong,et al.  Mesh segmentation for parallel decompression on GPU , 2012, CVM'12.

[7]  Justus Thies,et al.  Face2Face: Real-Time Face Capture and Reenactment of RGB Videos , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Justus Thies,et al.  Demo of Face2Face: real-time face capture and reenactment of RGB videos , 2016, SIGGRAPH Emerging Technologies.

[9]  Hans-Peter Seidel,et al.  Exchanging Faces in Images , 2004, Comput. Graph. Forum.

[10]  Jihun Yu,et al.  Realtime facial animation with on-the-fly correctives , 2013, ACM Trans. Graph..

[11]  Sami Romdhani,et al.  A 3D Face Model for Pose and Illumination Invariant Face Recognition , 2009, 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance.

[12]  Hanspeter Pfister,et al.  Face transfer with multilinear models , 2005, SIGGRAPH 2005.

[13]  Jia-Kai Chou,et al.  Face-off: automatic alteration of facial features , 2012, Multimedia Tools and Applications.

[14]  Olga Sorkine-Hornung,et al.  Mixed Finite Elements for Variational Surface Modeling , 2010, Comput. Graph. Forum.

[15]  Tal Hassner,et al.  Regressing Robust and Discriminative 3D Morphable Models with a Very Deep Neural Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Shree K. Nayar,et al.  Face swapping: automatically replacing faces in photographs , 2008, SIGGRAPH 2008.

[17]  Kun Zhou,et al.  3D shape regression for real-time facial animation , 2013, ACM Trans. Graph..

[18]  Ira Kemelmacher-Shlizerman,et al.  Transfiguring portraits , 2016, ACM Trans. Graph..

[19]  Tomaso A. Poggio,et al.  Reanimating Faces in Images and Video , 2003, Comput. Graph. Forum.

[20]  Xiaogang Jin,et al.  Enhancing the Symmetry and Proportion of 3D Face Geometry , 2012, IEEE Transactions on Visualization and Computer Graphics.

[21]  Yuan Lin,et al.  Face replacement with large-pose differences , 2012, ACM Multimedia.

[22]  Anderson Rocha,et al.  Manifold Learning and Spectral Clustering for Image Phylogeny Forests , 2016, IEEE Transactions on Information Forensics and Security.

[23]  Thomas Vetter,et al.  A morphable model for the synthesis of 3D faces , 1999, SIGGRAPH.

[24]  Kun Zhou,et al.  Real-time facial animation with image-based dynamic avatars , 2016, ACM Trans. Graph..

[25]  Jianping Hu,et al.  Least-squares images for edge-preserving smoothing , 2015, Computational Visual Media.

[26]  Kun Zhou,et al.  Parametric Reshaping of Portrait Images for Weight-change , 2018, IEEE Computer Graphics and Applications.

[27]  Khaled F. Hussain,et al.  Video face replacement system using a modified Poisson blending technique , 2014, 2014 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS).

[28]  Anil K. Jain,et al.  Unconstrained Face Recognition: Identifying a Person of Interest From a Media Collection , 2014, IEEE Transactions on Information Forensics and Security.

[29]  Tamy Boubekeur,et al.  GeoBrush: Interactive Mesh Geometry Cloning , 2011, Comput. Graph. Forum.

[30]  Yiying Tong,et al.  FaceWarehouse: A 3D Facial Expression Database for Visual Computing , 2014, IEEE Transactions on Visualization and Computer Graphics.

[31]  Jiancheng Lv,et al.  A surgical simulation system for predicting facial soft tissue deformation , 2016, Computational Visual Media.

[32]  Yuan Lin,et al.  Face Swapping under Large Pose Variations: A 3D Model Based Approach , 2012, 2012 IEEE International Conference on Multimedia and Expo.

[33]  Josephine Sullivan,et al.  One millisecond face alignment with an ensemble of regression trees , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[34]  William J. Christmas,et al.  A Multiresolution 3D Morphable Face Model and Fitting Framework , 2016, VISIGRAPP.

[35]  Hao Li,et al.  Real-Time Facial Segmentation and Performance Capture from RGB Input , 2016, ECCV.

[36]  Jie Lv,et al.  A Face Replacement System Based on 3D Face Model , 2015 .