Sketch-based Face Editing in Video Using Identity Deformation Transfer

We address the problem of using hand-drawn sketch to edit facial identity, such as enlarging the shape or modifying the position of eyes or mouth, in the whole video. This task is formulated as a 3D face model reconstruction and deformation problem. We first introduce a two-stage real-time 3D face model fitting schema to recover facial identity and expressions from the video. We recognize the user's editing intention from the input sketch as a set of facial modifications. A novel identity deformation algorithm is then proposed to transfer these deformations from 2D space to 3D facial identity directly, while preserving the facial expressions. Finally, these changes are propagated to the whole video with the modified identity. Experimental results demonstrate that our method can effectively edit facial identity in video based on the input sketch with high consistency and fidelity.

[1]  Yu Tian,et al.  Learning to Forecast and Refine Residual Motion for Image-to-Video Generation , 2018, ECCV.

[2]  Timothy F. Cootes,et al.  Active Appearance Models , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Justus Thies,et al.  Face2Face: real-time face capture and reenactment of RGB videos , 2019, Commun. ACM.

[4]  Yiying Tong,et al.  FaceWarehouse: A 3D Facial Expression Database for Visual Computing , 2014, IEEE Transactions on Visualization and Computer Graphics.

[5]  Moon-Ryul Jung,et al.  Local shape blending using coherent weighted regions , 2011, The Visual Computer.

[6]  Mahadev Satyanarayanan,et al.  OpenFace: A general-purpose face recognition library with mobile applications , 2016 .

[7]  Thomas Vetter,et al.  A morphable model for the synthesis of 3D faces , 1999, SIGGRAPH.

[8]  Oswald Aldrian,et al.  Inverse Rendering of Faces with a 3D Morphable Model , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Alla Sheffer,et al.  Modeling from contour drawings , 2009, SBIM '09.

[10]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[11]  Adam Finkelstein,et al.  Suggestive contours for conveying shape , 2003, ACM Trans. Graph..

[12]  Zhigang Deng,et al.  Interactive 3D facial expression posing through 2D portrait manipulation , 2008, Graphics Interface.

[13]  C. Lawrence Zitnick,et al.  Edge Boxes: Locating Object Proposals from Edges , 2014, ECCV.

[14]  Xin Tong,et al.  Automatic acquisition of high-fidelity facial performances using monocular videos , 2014, ACM Trans. Graph..

[15]  Yichen Wei,et al.  Learning best views of 3D shapes from sketch contour , 2015, The Visual Computer.

[16]  Fei Yang,et al.  Facial expression editing in video using a temporally-smooth factorization , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Odest Chadwicke Jenkins,et al.  Sketching articulation and pose for facial animation , 2006, SCA '06.

[18]  João Orvalho,et al.  Sketch express: A sketching interface for facial animation , 2012, Comput. Graph..

[19]  Mario Costa Sousa,et al.  Sketch-based modeling: A survey , 2009, Comput. Graph..

[20]  William J. Christmas,et al.  A Multiresolution 3D Morphable Face Model and Fitting Framework , 2016, VISIGRAPP.

[21]  Simon Lucey,et al.  Deformable Model Fitting by Regularized Landmark Mean-Shift , 2010, International Journal of Computer Vision.

[22]  Jovan Popović,et al.  Deformation transfer for triangle meshes , 2004, SIGGRAPH 2004.

[23]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Van Nostrand,et al.  Error Bounds for Convolutional Codes and an Asymptotically Optimum Decoding Algorithm , 1967 .

[25]  Markus Gross,et al.  Deformation Transfer for Detail-Preserving Surface Editing , 2006 .

[26]  Jihun Yu,et al.  Realtime facial animation with on-the-fly correctives , 2013, ACM Trans. Graph..

[27]  Yichen Wei,et al.  Sketch-Based Retrieval Using Content-Aware Hashing , 2014, PCM.

[28]  Jian Sun,et al.  Face Alignment at 3000 FPS via Regressing Local Binary Features , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[29]  Xin Tong,et al.  Accurate and Robust 3D Facial Capture Using a Single RGBD Camera , 2013, 2013 IEEE International Conference on Computer Vision.

[30]  Scott Schaefer,et al.  Image deformation using moving least squares , 2006, ACM Trans. Graph..

[31]  Lanfen Lin,et al.  A new sketch-based 3D model retrieval approach by using global and local features , 2014, Graph. Model..

[32]  Fei Yang,et al.  Expression flow for 3D-aware face component transfer , 2011, SIGGRAPH 2011.

[33]  Yu Tian,et al.  CR-GAN: Learning Complete Representations for Multi-view Generation , 2018, IJCAI.

[34]  Kun Zhou,et al.  Displaced dynamic expression regression for real-time facial tracking and animation , 2014, ACM Trans. Graph..

[35]  Chi-Ho Chan,et al.  Face Recognition Using a Unified 3D Morphable Model , 2016, ECCV.

[36]  Fang Wang,et al.  Sketch-based 3D shape retrieval using Convolutional Neural Networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Petros Faloutsos,et al.  Bringing Sketch Recognition into Your Hands , 2011, IEEE Computer Graphics and Applications.

[38]  Jovan Popović,et al.  Semantic deformation transfer , 2009, SIGGRAPH 2009.

[39]  H. Shum,et al.  Subspace gradient domain mesh deformation , 2006, SIGGRAPH 2006.

[40]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[41]  Alan Brunton,et al.  Multilinear Wavelets: A Statistical Shape Space for Human Faces , 2014, ECCV.

[42]  Feng Xu,et al.  Controllable high-fidelity facial performance transfer , 2014, ACM Trans. Graph..

[43]  Jihun Yu,et al.  Unconstrained realtime facial performance capture , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Jan Kautz,et al.  MoCoGAN: Decomposing Motion and Content for Video Generation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[45]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[46]  Thabo Beeler,et al.  Real-time high-fidelity facial performance capture , 2015, ACM Trans. Graph..

[47]  Derek Bradley,et al.  An anatomically-constrained local deformation model for monocular face capture , 2016, ACM Trans. Graph..

[48]  Marcus A. Magnor,et al.  Sparse localized deformation components , 2013, ACM Trans. Graph..

[49]  Harry Shum,et al.  Face poser: Interactive modeling of 3D facial expressions using facial priors , 2009, TOGS.

[50]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[51]  Yizhou Yu,et al.  DeepSketch2Face , 2017, ACM Trans. Graph..

[52]  Baining Guo,et al.  3D cartoon face generation by local deformation mapping , 2016, The Visual Computer.