Practice and Theory of Blendshape Facial Models

Blendshapes”, a simple linear model of facial expression, is the prevalent approach to realistic facial animation. It has driven animated characters in Hollywood films, and is a standard feature of commercial animation packages. The blendshape approach originated in industry, and became a subject of academic research relatively recently. This survey describes the published state of the art in this area, covering both literature from the graphics research community, and developments published in industry forums. We show that, despite the simplicity of the blendshape approach, there remain open problems associated with this fundamental technique.

[1]  Susan Zwerman,et al.  The VES Handbook of Visual Effects : Industry Standard VFX Practices and Procedures , 2014 .

[2]  John P. Lewis,et al.  Probable and Improbable Faces , 2014 .

[3]  Jun-yong Noh,et al.  A Survey of Facial Modeling and Animation Techniques , 2001 .

[4]  Katsu Yamane,et al.  Natural Motion Animation through Constraining and Deconstraining at Will , 2003, IEEE Trans. Vis. Comput. Graph..

[5]  Norbert Krüger,et al.  Face Recognition by Elastic Bunch Graph Matching , 1997, CAIP.

[6]  Yangang Wang,et al.  Online modeling for realtime facial animation , 2013, ACM Trans. Graph..

[7]  Paul Debevec,et al.  A blendshape model that incorporates physical interaction , 2012, Comput. Animat. Virtual Worlds.

[8]  Wan-Chun Ma,et al.  Comprehensive Facial Performance Capture , 2011, Comput. Graph. Forum.

[9]  Frederic I. Parke,et al.  Control Parameterization for Facial Animation , 1991 .

[10]  David Komorowski,et al.  A hybrid approach to facial rigging , 2010, SA '10.

[11]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[12]  Keith Waters,et al.  Computer facial animation , 1996 .

[13]  Hyeong-Seok Ko,et al.  Performance-driven muscle-based facial animation , 2001, Comput. Animat. Virtual Worlds.

[14]  Fernando De la Torre,et al.  Interactive region-based linear 3D face models , 2011, SIGGRAPH 2011.

[15]  Zoran Popovic,et al.  The space of human body shapes: reconstruction and parameterization from range scans , 2003, ACM Trans. Graph..

[16]  Ken-ichi Anjyo,et al.  Spacetime expression cloning for blendshapes , 2012, TOGS.

[17]  John P. Lewis,et al.  Universal capture - image-based facial animation for "The Matrix Reloaded" , 2005, SIGGRAPH Courses.

[18]  Hanspeter Pfister,et al.  Face transfer with multilinear models , 2005, SIGGRAPH 2005.

[19]  Frederick I. Parke,et al.  Computer generated animation of faces , 1972, ACM Annual Conference.

[20]  Jun-yong Noh,et al.  Expression cloning , 2001, SIGGRAPH 2001.

[21]  Jihun Yu,et al.  Realtime facial animation with on-the-fly correctives , 2013, ACM Trans. Graph..

[22]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[23]  Sami Romdhani,et al.  Optimal Step Nonrigid ICP Algorithms for Surface Registration , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Wojciech Matusik,et al.  Multi-scale capture of facial geometry and motion , 2007, ACM Trans. Graph..

[25]  J. P. Lewis,et al.  Identifying salient points , 2009, SIGGRAPH ASIA '09.

[26]  Baining Guo,et al.  Geometry-driven photorealistic facial expression synthesis , 2003, IEEE Transactions on Visualization and Computer Graphics.

[27]  Julien Mairal,et al.  Convex optimization with sparsity-inducing norms , 2011 .

[28]  Chong Gu.  Smoothing Spline Anova Models , 2002 .

[29]  Hao Li,et al.  Example-based facial rigging , 2010, ACM Transactions on Graphics.

[30]  Verónica Orvalho,et al.  A Facial Rigging Survey , 2012, Eurographics.

[31]  John P. Lewis,et al.  Universal capture: image-based facial animation for "The Matrix Reloaded" , 2003, SIGGRAPH '03.

[32]  Zhigang Deng,et al.  Computer Facial Animation: A Survey , 2008 .

[33]  Tsuneya Kurihara,et al.  Modeling deformable human hands from medical images , 2004, SCA '04.

[34]  Thabo Beeler,et al.  High-quality single-shot capture of facial geometry , 2010, ACM Trans. Graph..

[35]  Jovan Popović,et al.  Mesh-based inverse kinematics , 2005, SIGGRAPH 2005.

[36]  Kun Zhou,et al.  Displaced dynamic expression regression for real-time facial tracking and animation , 2014, ACM Trans. Graph..

[37]  João Orvalho,et al.  Sketch express: facial expressions made easy , 2011, SBIM '11.

[38]  Christoph Bregler,et al.  Facial expression space learning , 2002, 10th Pacific Conference on Computer Graphics and Applications, 2002. Proceedings..

[39]  Hyeong-Seok Ko,et al.  Analysis and synthesis of facial expressions with hand-generated muscle actuation basis , 2005, SIGGRAPH Courses.

[40]  J. Garcke.  Sparse Grid Tutorial , 2007 .

[41]  Marcus A. Magnor,et al.  Sparse localized deformation components , 2013, ACM Trans. Graph..

[42]  Verónica Orvalho,et al.  Sketch-Based Controllers for Blendshape Facial Animation , 2015, Eurographics.

[43]  Markus H. Gross,et al.  Pose-space animation and transfer of facial details , 2008, SCA '08.

[44]  Peter Eisert,et al.  Fast nonrigid mesh registration with a data-driven deformation prior , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[45]  Christian Theobalt,et al.  Reconstructing detailed dynamic face geometry from monocular video , 2013, ACM Trans. Graph..

[46]  Thabo Beeler,et al.  Facial performance enhancement using dynamic shape space analysis , 2014, TOGS.

[47]  Wan-Chun Ma,et al.  A facial composite editor for blendshape characters , 2012, DigiPro '12.

[48]  Frederic I. Parke,et al.  A parametric model for human faces. , 1974 .

[49]  Henrik Wann Jensen.  Digital face cloning , 2003, SIGGRAPH '03.

[50]  Zhigang Deng,et al.  Animating blendshape faces by cross-mapping motion capture data , 2006, I3D '06.

[51]  Ken-ichi Anjyo,et al.  Scattered data interpolation for computer graphics , 2014, SIGGRAPH '14.

[52]  Mark Sagar.  Facial performance capture and expressive translation for King Kong , 2006, SIGGRAPH Sketches.

[53]  David P. Dobkin,et al.  Multiresolution mesh morphing , 1999, SIGGRAPH.

[54]  Demetri Terzopoulos,et al.  Physically-based facial modelling, analysis, and animation , 1990, Comput. Animat. Virtual Worlds.

[55]  Thomas Vetter,et al.  A morphable model for the synthesis of 3D faces , 1999, SIGGRAPH.

[56]  Yuting Ye,et al.  High fidelity facial animation capture and retargeting with contours , 2013, SCA '13.

[57]  Derek Bradley,et al.  High resolution passive facial performance capture , 2010, ACM Trans. Graph..

[58]  Geoffrey Irving,et al.  Compression and Direct Manipulation of Complex Blendshape Models , 2011 .

[59]  Dimitris N. Metaxas,et al.  The integration of optical flow and deformable models with applications to human face shape and motion estimation , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[60]  John P. Lewis,et al.  Reducing blendshape interference by selected motion attenuation , 2005, I3D '05.

[61]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[62]  Jun Saito,et al.  Smooth contact-aware facial blendshapes transfer , 2013, DigiPro.

[63]  Hyun-Jin Kim Paul,et al.  Facial Retargeting by Adding Supplemental Blendshapes = 블렌드쉐입을 추가하는 방법을 이용한 페이셜 리타게팅 , 2011 .

[64]  Wojciech Matusik,et al.  Video face replacement , 2011, ACM Trans. Graph..

[65]  Luc Van Gool,et al.  Face/Off: live facial puppetry , 2009, SCA '09.

[66]  David Salesin,et al.  Resynthesizing facial animation through 3D model-based tracking , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[67]  Jovan Popović,et al.  Deformation transfer for triangle meshes , 2004, SIGGRAPH 2004.

[68]  Steven M. Seitz,et al.  Spacetime faces , 2004, ACM Trans. Graph..

[69]  R. Caflisch.  Monte Carlo and quasi-Monte Carlo methods , 1998, Acta Numerica.

[70]  Yeongho Seol,et al.  Artist friendly facial animation retargeting , 2011, ACM Trans. Graph..

[71]  Pieter Peers,et al.  Facial performance synthesis using deformation-driven polynomial displacement maps , 2008, SIGGRAPH Asia '08.

[72]  John P. Lewis,et al.  Ripple-free local bases by design , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[73]  Sham Tickoo.  Autodesk Maya 2010: A Comprehensive Guide , 2009 .

[74]  Ken-ichi Anjyo,et al.  Direct Manipulation Blendshapes , 2010, IEEE Computer Graphics and Applications.

[75]  Jason Osipa.  Stop Staring: Facial Modeling and Animation Done Right , 2003 .

[76]  Taehyun Rhee,et al.  Real-time facial animation from live video tracking , 2011, SCA '11.

[77]  Tomaso A. Poggio,et al.  Reanimating Faces in Images and Video , 2003, Comput. Graph. Forum.

[78]  Sebastian Thrun,et al.  SCAPE: shape completion and animation of people , 2005, SIGGRAPH 2005.

[79]  Lance Williams,et al.  Human face project , 2005, SIGGRAPH Courses.

[80]  Jing Xiao,et al.  A Closed-Form Solution to Non-rigid Shape and Motion Recovery , 2004, ECCV.

[81]  Hao Li,et al.  Realtime performance-based facial animation , 2011, ACM Trans. Graph..

[82]  Ronald Fedkiw,et al.  Automatic determination of facial muscle activations from sparse motion capture marker data , 2005, ACM Trans. Graph..

[83]  Taehyun Rhee,et al.  Principal Component Analysis and Laplacian Splines: Steps Toward a Unified Model , 2014 .

[84]  Yeongho Seol,et al.  Weighted pose space editing for facial animation , 2012, The Visual Computer.

[85]  Pieter Peers,et al.  Facial cartography: interactive high-resolution scan correspondence , 2011, SIGGRAPH '11.

[86]  Henrique S. Malvar,et al.  Making Faces , 2019, Topoi.

[87]  Kun Zhou,et al.  3D shape regression for real-time facial animation , 2013, ACM Trans. Graph..

[88]  Erika Chuang,et al.  Performance Driven Facial Animation using Blendshape Interpolation , 2002 .

[89]  Derek Bradley,et al.  Rigid stabilization of facial expressions , 2014, ACM Trans. Graph..

[90]  Barbara Flueckiger,et al.  Computer-Generated Characters in Avatar and Benjamin Button , 2011 .

[91]  Mark Meyer,et al.  Key Point Subspace Acceleration and soft caching , 2007, SIGGRAPH 2007.

[92]  Harry Shum,et al.  Face poser: Interactive modeling of 3D facial expressions using facial priors , 2009, TOGS.

[93]  John P. Lewis,et al.  Perceiving Visual Emotions with Speech , 2006, IVA.

[94]  Derek Bradley,et al.  High-quality passive facial performance capture using anchor frames , 2011, ACM Trans. Graph..

[95]  John P. Lewis,et al.  Pose Space Deformation: A Unified Approach to Shape Interpolation and Skeleton-Driven Deformation , 2000, SIGGRAPH.

[96]  Ken-ichi Anjyo,et al.  A Practical Approach to Direct Manipulation Blendshapes , 2012, J. Graph. Tools.

[97]  Zhigang Deng,et al.  Orthogonal-Blendshape-Based Editing System for Facial Motion Capture Data , 2008, IEEE Computer Graphics and Applications.

[98]  Markus Gross,et al.  Deformation Transfer for Detail-Preserving Surface Editing , 2006 .

[99]  Jing Xiao,et al.  Vision-based control of 3D facial animation , 2003, SCA '03.

[100]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[101]  P. Ekman,et al.  What the face reveals : basic and applied studies of spontaneous expression using the facial action coding system (FACS) , 2005 .

[102]  Michael Elad,et al.  Sparse and Redundant Representations - From Theory to Applications in Signal and Image Processing , 2010 .

[103]  Lance Williams,et al.  Performance-driven facial animation , 1990, SIGGRAPH Courses.

[104]  Frédéric H. Pighin,et al.  Unsupervised learning for speech motion editing , 2003, SCA '03.

[105]  Gene S. Lee,et al.  Practical experiences with pose space deformation , 2009, SIGGRAPH ASIA '09.

[106]  Jianlin Xia,et al.  Fast algorithms for hierarchically semiseparable matrices , 2010, Numer. Linear Algebra Appl..