High fidelity facial animation capture and retargeting with contours

Human beings are naturally sensitive to subtle cues in facial expressions, especially in areas of the eyes and mouth. Current facial motion capture methods fail to accurately reproduce motions in those areas due to multiple limitations. In this paper, we present a new performance capture method that focuses on the perceptually important contour features on the face. Additionally, the output of our two-step optimization scheme is also easily editable by an animator. To illustrate the strength of our system, we present a retargeting application that incorporates primary contour lines to map a performance with lip-sync from an actor to a creature.

[1]  Luc Van Gool,et al.  Face/Off: live facial puppetry , 2009, SCA '09.

[2]  Mark Meyer,et al.  Discrete Differential-Geometry Operators for Triangulated 2-Manifolds , 2002, VisMath.

[3]  Heinrich H. Bülthoff,et al.  Manipulating Video Sequences to Determine the Components of Conversational Facial Expressions , 2005, TAP.

[4]  John P. Lewis,et al.  Compression and direct manipulation of complex blendshape models , 2011, ACM Trans. Graph..

[5]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[6]  Hans Westman,et al.  SIGGRAPH Asia , 2007, COMG.

[7]  João Orvalho,et al.  Sketch express: facial expressions made easy , 2011, SBIM '11.

[8]  Song Zhang,et al.  High-Resolution, Real-time 3D Shape Acquisition , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[9]  Paul E. Debevec,et al.  Multiview face capture using polarized spherical gradient illumination , 2011, ACM Trans. Graph..

[10]  H. Bülthoff,et al.  The contribution of different facial regions to the recognition of conversational expressions. , 2008, Journal of vision.

[11]  Ken-ichi Anjyo,et al.  Spacetime expression cloning for blendshapes , 2012, TOGS.

[12]  Demetri Terzopoulos,et al.  Analysis and Synthesis of Facial Image Sequences Using Physical and Anatomical Models , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  P. Ekman.  Facial expression and emotion. , 1993, The American psychologist.

[14]  Mark Sagar.  Facial performance capture and expressive translation for King Kong , 2006, SIGGRAPH Sketches.

[15]  Olga Sorkine-Hornung,et al.  On Linear Variational Surface Deformation Methods , 2008, IEEE Transactions on Visualization and Computer Graphics.

[16]  Wan-Chun Ma,et al.  Comprehensive Facial Performance Capture , 2011, Comput. Graph. Forum.

[17]  Jihun Yu,et al.  Realtime facial animation with on-the-fly correctives , 2013, ACM Trans. Graph..

[18]  Christoph Bregler,et al.  Eigen-points [image matching] , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[19]  Hao Li,et al.  Realtime performance-based facial animation , 2011, ACM Trans. Graph..

[20]  Andrew Blake,et al.  Separability of pose and expression in facial tracking and animation , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[21]  John P. Lewis,et al.  Universal capture: image-based facial animation for "The Matrix Reloaded" , 2003, SIGGRAPH '03.

[22]  Ronald Fedkiw,et al.  Automatic determination of facial muscle activations from sparse motion capture marker data , 2005, ACM Trans. Graph..

[23]  Derek Bradley,et al.  High resolution passive facial performance capture , 2010, ACM Trans. Graph..

[24]  Christoph Bregler,et al.  ILM's multitrack: a new visual tracking framework for high-end VFX production , 2009, SIGGRAPH '09.

[25]  Leonidas J. Guibas,et al.  Robust single-view geometry and motion reconstruction , 2009, ACM Trans. Graph..

[26]  Harry Shum,et al.  Face poser: Interactive modeling of 3D facial expressions using facial priors , 2009, TOGS.

[27]  Erika Chuang,et al.  Performance Driven Facial Animation using Blendshape Interpolation , 2002 .

[28]  Mark Pauly,et al.  Example-based facial rigging , 2010, SIGGRAPH 2010.

[29]  Pieter Peers,et al.  Facial performance synthesis using deformation-driven polynomial displacement maps , 2008, SIGGRAPH Asia '08.

[30]  Wojciech Matusik,et al.  Multi-scale capture of facial geometry and motion , 2007, ACM Trans. Graph..

[31]  Wan-Chun Ma,et al.  The Digital Emily Project: Achieving a Photorealistic Digital Actor , 2010, IEEE Computer Graphics and Applications.

[32]  Jinxiang Chai,et al.  Leveraging motion capture and 3D scanning for high-fidelity facial performance acquisition , 2011, SIGGRAPH 2011.

[33]  Leonidas J. Guibas,et al.  Robust single-view geometry and motion reconstruction , 2009, SIGGRAPH 2009.

[34]  Derek Bradley,et al.  High-quality passive facial performance capture using anchor frames , 2011, ACM Trans. Graph..

[35]  Timothy F. Cootes,et al.  Active Appearance Models , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  Peter Robinson,et al.  3D Constrained Local Model for rigid and non-rigid facial tracking , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[37]  Jan Staes,et al.  ‘The Matrix Reloaded': A review of expert knowledge use for mapping ecosystem services , 2015 .

[38]  Michael J. Black,et al.  Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion , 1995, Proceedings of IEEE International Conference on Computer Vision.

[39]  Hao Li,et al.  Example-based facial rigging , 2010, ACM Transactions on Graphics.

[40]  Verónica Orvalho,et al.  A Facial Rigging Survey , 2012, Eurographics.

[41]  David Salesin,et al.  Resynthesizing facial animation through 3D model-based tracking , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[42]  Jovan Popović,et al.  Deformation transfer for triangle meshes , 2004, SIGGRAPH 2004.

[43]  Steven M. Seitz,et al.  Spacetime faces , 2004, ACM Trans. Graph..

[44]  Yong Yu,et al.  Facial animation by optimized blendshapes from motion capture data , 2008 .

[45]  Lance Williams,et al.  Performance-driven facial animation , 1990, SIGGRAPH Courses.