Modeling and Animating Realistic Faces from Images

We present a new set of techniques for modeling and animating realistic faces from photographs and videos. Given a set of face photographs taken simultaneously, our modeling technique allows the interactive recovery of a textured 3D face model. By repeating this process for several facial expressions, we acquire a set of face models that can be linearly combined to express a wide range of expressions. Given a video sequence, this linear face model can be used to estimate the face position, orientation, and facial expression at each frame. We illustrate these techniques on several datasets and demonstrate robust estimations of detailed face geometry and motion.

[1]  Tomaso A. Poggio,et al.  Hierarchical morphable models , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[2]  David Banks,et al.  Interactive shape metamorphosis , 1995, I3D '95.

[3]  Henrique S. Malvar,et al.  Making Faces , 2019, Topoi.

[4]  Tony Ezzat,et al.  MikeTalk: a talking facial display based on morphing visemes , 1998, Proceedings Computer Animation '98 (Cat. No.98EX169).

[5]  Frederick I. Parke,et al.  Computer gernerated animation of faces , 1998 .

[6]  Markus H. Gross,et al.  Simulating facial surgery using finite element models , 1996, SIGGRAPH.

[7]  Pascal Fua,et al.  From Regular Images to Animated Heads: A Least Squares Approach , 1998, ECCV.

[8]  F. I. Parke June,et al.  Computer Generated Animation of Faces , 1972 .

[9]  Demetri Terzopoulos,et al.  Realistic modeling for facial animation , 1995, SIGGRAPH.

[10]  Jitendra Malik,et al.  Recovering photometric properties of architectural scenes from photographs , 1998, SIGGRAPH.

[11]  Keith Waters,et al.  Computer facial animation , 1996 .

[12]  Thaddeus Beier,et al.  Feature-based image metamorphosis , 1992, SIGGRAPH.

[13]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[14]  Charles L. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[15]  P. Ekman,et al.  Unmasking the face : a guide to recognizing emotions from facial clues , 1975 .

[16]  Gregory D. Hager,et al.  Real-time tracking of image regions with changes in geometry and illumination , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[17]  Yasuhiko Watanabe,et al.  A trigonal prism-based method for hair image generation , 1992, IEEE Computer Graphics and Applications.

[18]  Irfan Essa,et al.  Head Tracking Using a Textured Polygonal Model , 1998 .

[19]  Lijun Yin,et al.  Constructing a 3D individualized head model from two orthogonal views , 1996, The Visual Computer.

[20]  F. A. Seiler,et al.  Numerical Recipes in C: The Art of Scientific Computing , 1989 .

[21]  Markus Gross,et al.  Simulating facial surgery using finite element models , 1996 .

[22]  Takaaki Akimoto,et al.  Automatic creation of 3D facial models , 1993, IEEE Computer Graphics and Applications.

[23]  O. Faugeras.  Three-dimensional computer vision: a geometric viewpoint , 1993 .

[24]  Timothy F. Cootes,et al.  Interpreting face images using active appearance models , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[25]  Marco La Cascia,et al.  Head tracking via robust registration in texture map images , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[26]  L. Vistnes.  The Artist??s Complete Guide to Facial Expression , 1992 .

[27]  Linda G. Shapiro,et al.  View-base Rendering: Visualizing Real Objects from Scanned Range and Color Data , 1997, Rendering Techniques.

[28]  Soren W. Henriksen,et al.  Manual of photogrammetry , 1980 .

[29]  Ken Perlin,et al.  An image synthesizer , 1988 .

[30]  Luc Van Gool,et al.  Active acquisition of 3D shape for moving objects , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[31]  Olivier D. Faugeras,et al.  Computing differential properties of 3-D shapes from stereoscopic images without 3-D models , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[32]  Demetri Terzopoulos,et al.  Snakes: Active contour models , 2004, International Journal of Computer Vision.

[33]  Peter W. Hallinan.  A low-dimensional representation of human faces for arbitrary lighting conditions , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[34]  Gregory M. Nielson,et al.  Scattered data modeling , 1993, IEEE Computer Graphics and Applications.

[35]  Hans Peter Graf,et al.  Robust face feature analysis for automatic speechreading and character animation , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[36]  David Salesin,et al.  Synthesizing realistic facial expressions from photographs , 1998, SIGGRAPH.

[37]  Kiyoharu Aizawa,et al.  Analysis and synthesis of facial image sequences in model-based image coding , 1994, IEEE Trans. Circuits Syst. Video Technol..

[38]  C. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[39]  Richard Szeliski,et al.  Recovering 3D Shape and Motion from Image Streams Using Nonlinear Least Squares , 1994, J. Vis. Commun. Image Represent..

[40]  Chil-Woo Lee,et al.  Automatic recognition of human facial expressions , 1995, Proceedings of IEEE International Conference on Computer Vision.

[41]  Federico Thomas,et al.  The illusion of life , 1981 .

[42]  Demetri Terzopoulos,et al.  Analysis and Synthesis of Facial Image Sequences Using Physical and Anatomical Models , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[43]  Gene H. Golub,et al.  Matrix computations , 1983 .

[44]  Sung Yong Shin,et al.  Image metamorphosis using snakes and free-form deformations , 1995, SIGGRAPH.

[45]  Pertti Roivainen,et al.  3-D Motion Estimation in Model-Based Facial Image Coding , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[46]  Dana H. Ballard,et al.  Computer Vision , 1982 .

[47]  Michael J. Black,et al.  Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion , 1995, Proceedings of IEEE International Conference on Computer Vision.

[48]  Gene H. Golub,et al.  Matrix computations (3rd ed.) , 1996 .

[49]  Tom Sanocki.  Pixar Animation Studios , 2003 .

[50]  Richard Szeliski,et al.  Creating full view panoramic image mosaics and texture-mapped models , 1997, International Conference on Computer Graphics and Interactive Techniques.

[51]  Wayne E. Carlson,et al.  Shape transformation for polyhedral objects , 1992, SIGGRAPH.

[52]  Richard Szeliski,et al.  Recovering 3D shape and motion from image streams using nonlinear least squares , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[53]  Sung Yong Shin,et al.  Image Metamorphosis with Scattered Feature Constraints , 1996, IEEE Trans. Vis. Comput. Graph..

[54]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[55]  Pat Hanrahan,et al.  Reflection from layered surfaces due to subsurface scattering , 1993, SIGGRAPH.

[56]  Michael W. Vannier,et al.  Three dimensional computer graphics for craniofacial surgical planning and evaluation , 1983, SIGGRAPH.

[57]  Frederic I. Parke,et al.  A parametric model for human faces. , 1974 .

[58]  Paul E. Debevec,et al.  Acquiring the reflectance field of a human face , 2000, SIGGRAPH.

[59]  Alex Pentland,et al.  Modeling, tracking and interactive animation of faces and heads//using input from video , 1996, Proceedings Computer Animation '96.

[60]  Tsuneya Kurihara,et al.  A Transformation Method for Modeling and Animation of the Human Face from Photographs , 1991 .

[61]  Dimitris N. Metaxas,et al.  Deformable model-based shape and motion analysis from images using motion residual error , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[62]  Kristinn R. Thórisson.  Gandalf: an embodied humanoid capable of real-time multimodal dialogue with people , 1997, AGENTS '97.

[63]  Michele Covell,et al.  Eigen-points: control-point location using principal component analyses , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[64]  Demetri Terzopoulos,et al.  Physically-based facial modelling, analysis, and animation , 1990, Comput. Animat. Virtual Worlds.

[65]  Thomas Vetter,et al.  A morphable model for the synthesis of 3D faces , 1999, SIGGRAPH.

[66]  Christoph Bregler,et al.  Video Rewrite: Driving Visual Speech with Audio , 1997, SIGGRAPH.

[67]  Steven M. Seitz,et al.  View morphing , 1996, SIGGRAPH.

[68]  Pascal Fua,et al.  3D face modeling from stereo and differential constraints , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[69]  Michael Isard,et al.  Active Contours: The Application of Techniques from Graphics, Vision, Control Theory and Statistics to Visual Tracking of Shapes in Motion , 2000 .

[70]  Richard Szeliski,et al.  Creating full view panoramic image mosaics and environment maps , 1997, SIGGRAPH.

[71]  Lance Williams,et al.  Performance-driven facial animation , 1990, SIGGRAPH.

[72]  Aaron F. Bobick,et al.  The direct computation of height from shading , 1991, Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[73]  Alex Pentland,et al.  Coding, Analysis, Interpretation, and Recognition of Facial Expressions , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[74]  SzeliskiRichard,et al.  Modeling and Animating Realistic Faces from Images , 2002 .

[75]  Thomas Vetter,et al.  Estimating Coloured 3D Face Models from Single Images: An Example Based Approach , 1998, ECCV.

[76]  Timothy F. Cootes,et al.  A unified approach to coding and interpreting face images , 1995, Proceedings of IEEE International Conference on Computer Vision.

[77]  Ken-ichi Anjyo,et al.  A simple method for extracting the natural beauty of hair , 1992, SIGGRAPH.

[78]  David Salesin,et al.  Resynthesizing facial animation through 3D model-based tracking , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[79]  William H. Press,et al.  The Art of Scientific Computing Second Edition , 1998 .

[80]  Paul Debevec,et al.  Modeling and Rendering Architecture from Photographs , 1996, SIGGRAPH 1996.