Realistic Facial Animation Using Image-Based 3D Morphing

We present new techniques for creating a realistic textured 3D facial model from several photographs of a human subject and for performing facial animation by morphing between models corresponding to di erent facial expressions. Starting from several uncalibrated views of an individual, we employ a user assisted technique to recover the camera poses corresponding to the views, as well as the 3D coordinates of a sparse set of chosen locations on the individuals face. A scattered data interpolation technique is then used to deform a generic face mesh into a 3D model of the individual's face. Having recovered the camera poses and the facial geometry, we extract from the input images a texture map for the model. An optical ow technique is used for improving the registration of the input images in texture space. This process is repeated for several facial expressions of a particular individual. To animate between these facial expressions we use 3D shape morphing between the corresponding facial models, while at the same time blending the corresponding textures. Using our technique we have been able to generate highly realistic facial models and natural looking transitions between di erent expressions. There is no landscape that we know as well as the human face. The twentyve-odd square inches containing the features is the most intimately scrutinized piece of territory in existence, examined constantly, and carefully, with far more than an intellectual interest. Every detail of the nose, eyes, and mouth, every

[1]  Frederick I. Parke,et al.  Computer generated animation of faces , 1972, ACM Annual Conference.

[2]  Frederic I. Parke,et al.  A parametric model for human faces. , 1974 .

[3]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[4]  Michael W. Vannier,et al.  Three dimensional computer graphics for craniofacial surgical planning and evaluation , 1983, SIGGRAPH.

[5]  Gene H. Golub,et al.  Matrix computations , 1983 .

[6]  Donald P. Greenberg,et al.  Improved Computational Methods for Ray Tracing , 1984, TOGS.

[7]  Keith Waters,et al.  A muscle model for animation three-dimensional facial expression , 1987, SIGGRAPH.

[8]  F. A. Seiler,et al.  Numerical Recipes in C: The Art of Scientific Computing , 1989 .

[9]  Demetri Terzopoulos,et al.  Physically-based facial modelling, analysis, and animation , 1990, Comput. Animat. Virtual Worlds.

[10]  Tsuneya Kurihara,et al.  A Transformation Method for Modeling and Animation of the Human Face from Photographs , 1991 .

[11]  P. Anandan,et al.  Hierarchical Model-Based Motion Estimation , 1992, ECCV.

[12]  Wayne E. Carlson,et al.  Shape transformation for polyhedral objects , 1992, SIGGRAPH.

[13]  Michael Gleicher,et al.  Through-the-lens camera control , 1992, SIGGRAPH.

[14]  Takaaki Akimoto,et al.  Automatic creation of 3D facial models , 1993, IEEE Computer Graphics and Applications.

[15]  Gregory M. Nielson,et al.  Scattered data modeling , 1993, IEEE Computer Graphics and Applications.

[16]  O. Faugeras.  Three-dimensional computer vision: a geometric viewpoint , 1993 .

[17]  Richard Szeliski,et al.  Recovering 3D Shape and Motion from Image Streams Using Nonlinear Least Squares , 1994, J. Vis. Commun. Image Represent..

[18]  Pascal Fua,et al.  Registration without correspondences , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[19]  David Banks,et al.  Interactive shape metamorphosis , 1995, I3D '95.

[20]  Timothy F. Cootes,et al.  A unified approach to coding and interpreting face images , 1995, Proceedings of IEEE International Conference on Computer Vision.

[21]  Demetri Terzopoulos,et al.  Realistic modeling for facial animation , 1995, SIGGRAPH.

[22]  Keith Waters,et al.  Computer facial animation , 1996 .

[23]  Steven M. Seitz,et al.  View morphing , 1996, SIGGRAPH.

[24]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[25]  Kristinn R. Thórisson.  Gandalf: an embodied humanoid capable of real-time multimodal dialogue with people , 1997, AGENTS '97.

[26]  Thaddeus Beier,et al.  Feature-based image metamorphosis , 1998 .