High-Quality Face Capture Using Anatomical Muscles

Muscle-based systems have the potential to provide both anatomical accuracy and semantic interpretability as compared to blendshape models; however, a lack of expressivity and differentiability has limited their impact. Thus, we propose modifying a recently developed rather expressive muscle-based system in order to make it fully-differentiable; in fact, our proposed modifications allow this physically robust and anatomically accurate muscle model to conveniently be driven by an underlying blendshape basis. Our formulation is intuitive, natural, as well as monolithically and fully coupled such that one can differentiate the model from end to end, which makes it viable for both optimization and learning-based approaches for a variety of applications. We illustrate this with a number of examples including both shape matching of three-dimensional geometry as as well as the automatic determination of a three-dimensional facial pose from a single two-dimensional RGB image without using markers or depth information.

[1]  Ronald Fedkiw,et al.  Finite volume methods for the simulation of skeletal muscle , 2003, SCA '03.

[2]  Ronald Fedkiw,et al.  Art-directed muscle simulation for high-end facial animation , 2016, Symposium on Computer Animation.

[3]  Justus Thies,et al.  Face2Face: real-time face capture and reenactment of RGB videos , 2019, Commun. ACM.

[4]  Yiying Tong,et al.  FaceWarehouse: A 3D Facial Expression Database for Visual Computing , 2014, IEEE Transactions on Visualization and Computer Graphics.

[5]  Nicolas Stoiber,et al.  Realtime Performance‐Driven Physical Simulation for Facial Animation , 2018, Comput. Graph. Forum.

[6]  Mark Pauly,et al.  Phace: physics-based face modeling and animation , 2017, ACM Trans. Graph..

[7]  Hao Li,et al.  Realtime performance-based facial animation , 2011, ACM Trans. Graph..

[8]  G. Strang.  VARIATIONAL CRIMES IN THE FINITE ELEMENT METHOD , 1972 .

[9]  Ronald Fedkiw,et al.  Automatic determination of facial muscle activations from sparse motion capture marker data , 2005, ACM Trans. Graph..

[10]  Sami Romdhani,et al.  A 3D Face Model for Pose and Illumination Invariant Face Recognition , 2009, 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance.

[11]  David Salesin,et al.  Resynthesizing facial animation through 3D model-based tracking , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[12]  Ronald Fedkiw,et al.  Fully automatic generation of anatomical face simulation models , 2015, Symposium on Computer Animation.

[13]  Derek Bradley,et al.  An anatomically-constrained local deformation model for monocular face capture , 2016, ACM Trans. Graph..

[14]  Derek Bradley,et al.  High-quality passive facial performance capture using anchor frames , 2011, ACM Trans. Graph..

[15]  Ronald Fedkiw,et al.  Invertible finite elements for robust simulation of large deformation , 2004, SCA '04.

[16]  Janne Heikkilä,et al.  A four-step camera calibration procedure with implicit image correction , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[17]  Manolis I. A. Lourakis,et al.  Is Levenberg-Marquardt the most efficient optimization algorithm for implementing bundle adjustment? , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[18]  Paul Debevec,et al.  The Light Stages and Their Applications to Photoreal Digital Actors , 2012, SIGGRAPH 2012.

[19]  Anatomical features of the incisivus labii superioris muscle and its relationships with the upper mucolabial fold, labial glands, and modiolar area , 2018, Scientific Reports.

[20]  Pushmeet Kohli,et al.  Real-Time Face Reconstruction from a Single Depth Image , 2014, 2014 2nd International Conference on 3D Vision.

[21]  Michael J. Black,et al.  Learning a model of facial shape and expression from 4D scans , 2017, ACM Trans. Graph..

[22]  Patrick Pérez,et al.  State of the Art on Monocular 3D Face Reconstruction, Tracking, and Applications , 2018, Comput. Graph. Forum.

[23]  Ronald Fedkiw,et al.  Creating and simulating skeletal muscle from the visible human data set , 2005, IEEE Transactions on Visualization and Computer Graphics.

[24]  Thomas Gerig,et al.  Gaussian Process Morphable Models , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Ronald Fedkiw,et al.  A new incompressibility discretization for a hybrid particle MAC grid representation with surface tension , 2015, J. Comput. Phys..

[26]  Ronald Fedkiw,et al.  Muscle simulation for facial animation in Kong: Skull Island , 2017, SIGGRAPH Talks.

[27]  Justus Thies,et al.  InverseFaceNet: Deep Monocular Inverse Face Rendering , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[28]  Derek Bradley,et al.  Enriching Facial Blendshape Rigs with Physical Simulation , 2017, Comput. Graph. Forum.

[29]  Nicolas Stoiber,et al.  BlendForces: A Dynamic Framework for Facial Animation , 2016, Comput. Graph. Forum.

[30]  Pat Hanrahan,et al.  An efficient representation for irradiance environment maps , 2001, SIGGRAPH.

[31]  Matthew Turk,et al.  A Morphable Model For The Synthesis Of 3D Faces , 1999, SIGGRAPH.

[32]  Kun Zhou,et al.  3D shape regression for real-time facial animation , 2013, ACM Trans. Graph..

[33]  Michael J. Black,et al.  Generating 3D faces using Convolutional Mesh Autoencoders , 2018, ECCV.

[34]  Marie-Paule Cani,et al.  Anatomy transfer , 2013, ACM Trans. Graph..

[35]  Merlin Nimier-David,et al.  Building and animating user-specific volumetric face rigs , 2016, Symposium on Computer Animation.

[36]  Xiaoming Liu,et al.  Large-Pose Face Alignment via CNN-Based Dense 3D Model Fitting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Ronald Fedkiw,et al.  Lessons from the evolution of an anatomical facial muscle model , 2017 .

[38]  Thabo Beeler,et al.  High-quality single-shot capture of facial geometry , 2010, ACM Trans. Graph..

[39]  James F. O'Brien,et al.  Fast and deep deformation approximations , 2018, ACM Trans. Graph..

[40]  Xi Zhou,et al.  Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network , 2018, ECCV.

[41]  Jaakko Lehtinen,et al.  Differentiable Monte Carlo ray tracing through edge sampling , 2018, ACM Trans. Graph..

[42]  F. Zajac.  Muscle and tendon: properties, models, scaling, and application to biomechanics and motor control. , 1989, Critical reviews in biomedical engineering.

[43]  Ken-ichi Anjyo,et al.  Practice and Theory of Blendshape Facial Models , 2014, Eurographics.

[44]  Andrew Jones,et al.  Multi‐View Stereo on Consistent Face Topology , 2017, Comput. Graph. Forum.

[45]  Xin Tong,et al.  Accurate and Robust 3D Facial Capture Using a Single RGBD Camera , 2013, 2013 IEEE International Conference on Computer Vision.

[46]  Georgios Tzimiropoulos,et al.  Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[47]  Ron Kimmel,et al.  Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[48]  Michael J. Black,et al.  OpenDR: An Approximate Differentiable Renderer , 2014, ECCV.

[49]  Ronald Fedkiw,et al.  Robust quasistatic finite elements and flesh simulation , 2005, SCA '05.

[50]  Stefanos Zafeiriou,et al.  A 3D Morphable Model Learnt from 10,000 Faces , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Yuting Ye,et al.  High fidelity facial animation capture and retargeting with contours , 2013, SCA '13.