Face Swapping: Realistic Image Synthesis Based on Facial Landmarks Alignment

We propose an image-based face swapping algorithm, which can be used to replace the face in the reference image with the same facial shape and features as the input face. First, a face alignment is made based on a group of detected facial landmarks, so that the aligned input face and the reference face are consistent in size and posture. Secondly, an image warping algorithm based on triangulation is presented to adjust the reference face and its background according to the aligned input faces. In order to achieve more accurate face swapping, a face parsing algorithm is introduced to realize the accurate detection of the face-ROIs, and then the face-ROI in the reference image is replaced with the input face-ROI. Finally, a Poisson image editing algorithm is adopted to realize the boundary processing and color correction between the replacement region and the original background, and then the final face swapping result is obtained. In the experiments, we compare our method with other face swapping algorithms and make a qualitative and quantitative analysis to evaluate the reality and the fidelity of the replaced face. The analysis results show that our method has some advantages in the overall performance of swapping effect.

[1]  Takeo Kanade,et al.  Face Alignment Refinement , 2015, 2015 IEEE Winter Conference on Applications of Computer Vision.

[2]  Honglak Lee,et al.  Object Contour Detection with a Fully Convolutional Encoder-Decoder Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Hans-Peter Seidel,et al.  Exchanging Faces in Images , 2004, Comput. Graph. Forum.

[4]  Shigeo Morishima,et al.  RSGAN: face swapping and editing using face and hair representation in latent spaces , 2018, SIGGRAPH Posters.

[5]  Enric Meinhardt,et al.  Poisson Image Editing , 2016, Image Process. Line.

[6]  Yuan Lin,et al.  Face Swapping under Large Pose Variations: A 3D Model Based Approach , 2012, 2012 IEEE International Conference on Multimedia and Expo.

[7]  Josephine Sullivan,et al.  One millisecond face alignment with an ensemble of regression trees , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Jakub Nalepa,et al.  Self-adaptive algorithm for segmenting skin regions , 2014, EURASIP J. Adv. Signal Process..

[9]  Rama Chellappa,et al.  Learning Common and Feature-Specific Patterns: A Novel Multiple-Sparse-Representation-Based Tracker , 2018, IEEE Transactions on Image Processing.

[10]  Sachit Mahajan,et al.  SwapItUp: A Face Swap Application for Privacy Protection , 2017, 2017 IEEE 31st International Conference on Advanced Information Networking and Applications (AINA).

[11]  Wojciech Matusik,et al.  Video face replacement , 2011, ACM Trans. Graph..

[12]  Chunhong Pan,et al.  Facial image composition based on active appearance model , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[13]  Neeta Nain,et al.  Fast Face Detection Based on Skin Segmentation and Facial Features , 2015, 2015 11th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS).

[14]  Jiasong Mu,et al.  Throat polyp detection based on compressed big data of voice with support vector machine algorithm , 2014, EURASIP Journal on Advances in Signal Processing.

[15]  Thomas S. Huang,et al.  Interactive Facial Feature Localization , 2012, ECCV.

[16]  Shree K. Nayar,et al.  Face swapping: automatically replacing faces in photographs , 2008, SIGGRAPH 2008.

[17]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[18]  Nikos Komodakis,et al.  Learning to compare image patches via convolutional neural networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Wu Bo,et al.  A Skin Segmentation Algorithm Based on Stacked Autoencoders , 2017, IEEE Transactions on Multimedia.

[20]  Lucas Theis,et al.  Fast Face-Swap Using Convolutional Neural Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[21]  Timothy F. Cootes,et al.  Active Appearance Models , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[22]  Tal Hassner,et al.  On Face Segmentation, Face Swapping, and Face Perception , 2017, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[23]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..