3D Magic Mirror: Automatic Video to 3D Caricature Translation

Caricature is an abstraction of a real person which distorts or exaggerates certain features, but still retains a likeness. While most existing works focus on 3D caricature reconstruction from 2D caricatures or translating 2D photos to 2D caricatures, this paper presents a real-time and automatic algorithm for creating expressive 3D caricatures with caricature style texture map from 2D photos or videos. To solve this challenging ill-posed reconstruction problem and cross-domain translation problem, we first reconstruct the 3D face shape for each frame, and then translate 3D face shape from normal style to caricature style by a novel identity and expression preserving VAE-CycleGAN. Based on a labeling formulation, the caricature texture map is constructed from a set of multi-view caricature images generated by CariGANs. The effectiveness and efficiency of our method are demonstrated by comparison with baseline implementations. The perceptual study shows that the 3D caricatures generated by our method meet people's expectations of 3D caricature style.

[1]  Ken-ichi Anjyo,et al.  Practice and Theory of Blendshape Facial Models , 2014, Eurographics.

[2]  Yiying Tong,et al.  FaceWarehouse: A 3D Facial Expression Database for Visual Computing , 2014, IEEE Transactions on Visualization and Computer Graphics.

[3]  William T. Freeman,et al.  Synthesizing Normalized Faces from Facial Identity Features , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Georgios Tzimiropoulos,et al.  Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[5]  Michael J. Black,et al.  Learning a model of facial shape and expression from 4D scans , 2017, ACM Trans. Graph..

[6]  Patrick Pérez,et al.  State of the Art on Monocular 3D Face Reconstruction, Tracking, and Applications , 2018, Comput. Graph. Forum.

[7]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[8]  Wen Gao,et al.  Semi‐Supervised Learning in Reconstructed Manifold Space for 3D Caricature Generation , 2009, Comput. Graph. Forum.

[9]  S. F. Wang,et al.  Manifold‐Based 3D Face Caricature Generation with Individualized Facial Feature Extraction , 2010, Comput. Graph. Forum.

[10]  Erik Reinhard,et al.  Human facial illustrations: Creation and psychophysical evaluation , 2004, TOGS.

[11]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[12]  Bailin Deng,et al.  3D Face Reconstruction With Geometry Details From a Single Image , 2017, IEEE Transactions on Image Processing.

[13]  Susan E. Brennan,et al.  From the Leonardo Archive , 2007, Leonardo.

[14]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[15]  Ron Kimmel,et al.  Computational caricaturization of surfaces , 2015, Comput. Vis. Image Underst..

[16]  Yaser Sheikh,et al.  Deep incremental learning for efficient high-fidelity face tracking , 2018, ACM Trans. Graph..

[17]  Jianfei Cai,et al.  CNN-Based Real-Time Dense Face Reconstruction with Inverse-Rendered Photo-Realistic Face Images , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Joaquim B. Cavalcante Neto,et al.  Three-Dimensional Face Caricaturing by Anthropometric Distortions , 2013, 2013 XXVI Conference on Graphics, Patterns and Images.

[19]  Olga Sorkine-Hornung,et al.  On Linear Variational Surface Deformation Methods , 2008, IEEE Transactions on Visualization and Computer Graphics.

[20]  Hanspeter Pfister,et al.  Face transfer with multilinear models , 2005, SIGGRAPH 2005.

[21]  A.J O'Toole,et al.  3D shape and 2D surface textures of human faces: the role of "averages" in attractiveness and age , 1999, Image Vis. Comput..

[22]  Lin Gao,et al.  Automatic unpaired shape deformation transfer , 2018, ACM Trans. Graph..

[23]  Thomas Vetter,et al.  A morphable model for the synthesis of 3D faces , 1999, SIGGRAPH.

[24]  Justus Thies,et al.  Real-time expression transfer for facial reenactment , 2015, ACM Trans. Graph..

[25]  Kun Zhou,et al.  CaricatureShop: Personalized and Photorealistic Caricature Sketching , 2018, IEEE Transactions on Visualization and Computer Graphics.

[26]  Yann LeCun,et al.  Dimensionality Reduction by Learning an Invariant Mapping , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[27]  Ergun Akleman,et al.  Making caricatures with morphing , 1997, SIGGRAPH '97.

[28]  Vladimir Kolmogorov,et al.  An experimental comparison of min-cut/max- flow algorithms for energy minimization in vision , 2001, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Yinghuan Shi,et al.  WebCaricature: a benchmark for caricature recognition , 2017, BMVC.

[30]  Olga Sorkine-Hornung,et al.  Geometric optimization via composite majorization , 2017, ACM Trans. Graph..

[31]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[32]  Kun Zhou,et al.  Warp-guided GANs for single-photo facial animation , 2018, ACM Trans. Graph..

[33]  Yizhou Yu,et al.  DeepSketch2Face , 2017, ACM Trans. Graph..

[34]  Harry Shum,et al.  Example-based caricature generation with exaggeration , 2002, 10th Pacific Conference on Computer Graphics and Applications, 2002. Proceedings..

[35]  Patrick Pérez,et al.  Deep video portraits , 2018, ACM Trans. Graph..

[36]  Min Chen,et al.  Automatic Generation of 3D Caricatures Based on Artistic Deformation Styles , 2011, IEEE Transactions on Visualization and Computer Graphics.

[37]  Paul W. H. Chung,et al.  Use of Neural Networks in Automatic Caricature Generation: An Approach Based on Drawing Style Capture , 2005, IbPRIA.

[38]  Jan Kautz,et al.  Multimodal Unsupervised Image-to-Image Translation , 2018, ECCV.

[39]  Patrick Pérez,et al.  MoFA: Model-Based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[40]  A. O'Toole,et al.  Three-Dimensional Caricatures of Human Heads: Distinctiveness and the Perception of Facial Age , 1997, Perception.

[41]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[42]  Amitabh Varshney,et al.  Montage4D: interactive seamless fusion of multiview video textures , 2018, I3D.

[43]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[44]  Yangang Wang,et al.  Online modeling for realtime facial animation , 2013, ACM Trans. Graph..

[45]  Wenjiang J. Fu.  Penalized Regressions: The Bridge versus the Lasso , 1998 .

[46]  Harry Shum,et al.  An efficient approach to learning inhomogeneous Gibbs model , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[47]  Jovan Popović,et al.  Deformation transfer for triangle meshes , 2004, SIGGRAPH 2004.

[48]  Jing Liao,et al.  CariGANs , 2018, ACM Trans. Graph..

[49]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  Davis E. King,et al.  Dlib-ml: A Machine Learning Toolkit , 2009, J. Mach. Learn. Res..

[51]  M. Zollhöfer,et al.  Self-Supervised Multi-level Face Model Learning for Monocular Reconstruction at Over 250 Hz , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[52]  Yu Qiao,et al.  A Discriminative Feature Learning Approach for Deep Face Recognition , 2016, ECCV.

[53]  Mark Pauly,et al.  Dynamic 3D avatar creation from hand-held video input , 2015, ACM Trans. Graph..

[54]  Lin Gao,et al.  Mesh-based Autoencoders for Localized Deformation Component Analysis , 2017, AAAI.

[55]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[56]  Christian Theobalt,et al.  Reconstruction of Personalized 3D Face Rigs from Monocular Video , 2016, ACM Trans. Graph..

[57]  Jianfei Cai,et al.  Alive Caricature from 2D to 3D , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[58]  Christian Rössl,et al.  Laplacian surface editing , 2004, SGP '04.

[59]  Kun Zhou,et al.  Displaced dynamic expression regression for real-time facial tracking and animation , 2014, ACM Trans. Graph..

[60]  J. Collomosse,et al.  4D video textures for interactive character appearance , 2014, Comput. Graph. Forum.

[61]  Habibollah Haron,et al.  Computer Generated Caricature: A Survey , 2010, 2010 International Conference on Cyberworlds.

[62]  Tsai-Yen Li,et al.  Automatic Caricature Generation by Analyzing Facial Features , 2004 .