Modeling, tracking and interactive animation of faces and heads//using input from video

We describe tools that use measurements from video for the extraction of facial modeling and animation parameters, head tracking, and real time interactive facial animation. These tools share common goals but rely on varying details of physical and geometric modeling and in their input measurement system. Accurate facial modeling involves fine details of geometry and muscle coarticulation. By coupling pixel by pixel measurements of surface motion to a physically based face model and a muscle control model, we have been able to obtain detailed spatio temporal records of both the displacement of each point on the facial surface and the muscle control required to produce the observed facial motion. We discuss the importance of this visually extracted representation in terms of realistic facial motion synthesis. A similar method that uses an ellipsoidal model of the head coupled with detailed estimates of visual motion allows accurate tracking of head motion in 3D. Additionally, by coupling sparse, fast visual measurements with our physically based model via an interpolation process, we have produced a real time interactive facial animation/mimicking system.

[1]  Norman I. Badler,et al.  Final Report to Nsf of the Standards for Facial Animation Workshop Final Report to Nsf of the Standards for Facial Animation Workshop , 1994 .

[2]  Demetri Terzopoulos,et al.  Modelling and animating faces using scanned data , 1991, Comput. Animat. Virtual Worlds.

[3]  Michael J. Black,et al.  Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion , 1995, Proceedings of IEEE International Conference on Computer Vision.

[4]  Alex Pentland,et al.  Probabilistic visual learning for object detection , 1995, Proceedings of IEEE International Conference on Computer Vision.

[5]  Eero P. Simoncelli.  Distributed representation and analysis of visual motion , 1993 .

[6]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[7]  Alex Pentland,et al.  Space-time gestures , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Demetri Terzopoulos,et al.  Physically-based facial modelling, analysis, and animation , 1990, Comput. Animat. Virtual Worlds.

[9]  Alex Pentland,et al.  Recovery of Nonrigid Motion and Structure , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  F. Girosi,et al.  Networks for approximation and learning , 1990, Proc. IEEE.

[11]  Richard Szeliski,et al.  Tracking with Kalman snakes , 1993 .

[12]  Lance Williams,et al.  Performance-driven facial animation , 1990, SIGGRAPH.

[13]  Alex Pentland,et al.  Motion regularization for model-based head tracking , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[14]  Irfan Essa,et al.  Tracking facial motion , 1994, Proceedings of 1994 IEEE Workshop on Motion of Non-rigid and Articulated Objects.

[15]  Edward H. Adelson,et al.  Layered representation for motion analysis , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Lance Williams,et al.  Animating images with drawings , 1994, SIGGRAPH.

[17]  Keith Waters,et al.  A muscle model for animation three-dimensional facial expression , 1987, SIGGRAPH.

[18]  Thomas S. Huang,et al.  Final Report To NSF of the Planning Workshop on Facial Expression Understanding , 1992 .

[19]  S. A. Wainwright,et al.  Mechanical Design in Organisms , 2020 .

[20]  Demetri Terzopoulos,et al.  Analysis and Synthesis of Facial Image Sequences Using Physical and Anatomical Models , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Frederic I. Parke,et al.  Techniques for facial animation , 1991 .

[22]  Demetri Terzopoulos,et al.  Realistic modeling for facial animation , 1995, SIGGRAPH.

[23]  Parke,et al.  Parameterized Models for Facial Animation , 1982, IEEE Computer Graphics and Applications.

[24]  Norman I. Badler,et al.  Animating facial expressions , 1981, SIGGRAPH '81.

[25]  P. Ekman.  Facial expressions of emotion: an old controversy and new findings. , 1992, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[26]  Irfan Essa,et al.  Analysis, interpretation and synthesis of facial expressions , 1995 .

[27]  Alex Pentland,et al.  Recursive estimation of structure and motion using relative orientation constraints , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[28]  Alex Pentland,et al.  Face recognition using view-based and modular eigenspaces , 1994, Optics & Photonics.

[29]  H. Yahia,et al.  Facial Animation With Muscle and Wrinkle Simulation , 1993 .

[30]  Alex Pentland,et al.  A vision system for observing and extracting facial action parameters , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[31]  Steven D. Pieper,et al.  Interactive graphics for plastic surgery: a task-level analysis and implementation , 1992, I3D '92.

[32]  Alex Pentland,et al.  View-based and modular eigenspaces for face recognition , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.