Compression of MPEG-4 facial animation parameters for transmission of talking heads

The emerging MPEG-4 standard supports the transmission and composition of facial animation with natural video. The new standard will include a facial animation parameter (FAP) set that is defined based on the study of minimal facial actions and is closely related to muscle actions. The FAP set enables model-based representation of natural or synthetic talking-head sequences and allows intelligible visual reproduction of facial expressions, emotions, and speech pronunciations at the receiver. This paper addresses the data-compression issue of talking heads and presents three methods for bit-rate reduction of FAPs. Compression efficiency is achieved by way of transform coding, principal component analysis, and FAP interpolation. These methods are independent of each other in nature and thus can be applied in combination to lower the bit-rate demand of FAPs, making possible the transmission of multiple talking heads over band-limited channels. The basic methods described here have been adopted into the MPEG-4 Visual Committee Draft and are readily applicable to other articulation data such as body animation parameters. The efficacy of the methods is demonstrated by both subjective and objective results.

[1]  Speech dialogue with facial displays , 1994, CHI '94.

[2]  Eric D. Petajan,et al.  MPEG-4 : Audio / Video & Synthetic Graphics / Audio for Mixed Media , 1998 .

[3]  Frederick I. Parke,et al.  Computer generated animation of faces , 1972, ACM Annual Conference.

[4]  Henrique S. Malvar,et al.  Making Faces , 2019, Topoi.

[5]  Lawrence Sirovich,et al.  Management and Analysis of Large Scientific Datasets , 1992 .

[6]  Frederick I. Parke,et al.  Computer gernerated animation of faces , 1998 .

[7]  Anil K. Jain.  Fundamentals of Digital Image Processing , 2018, Control of Color Imaging Systems.

[8]  Parke,et al.  Parameterized Models for Facial Animation , 1982, IEEE Computer Graphics and Applications.

[9]  Demetri Terzopoulos,et al.  Realistic modeling for facial animation , 1995, SIGGRAPH.

[10]  Fernando Pereira,et al.  MPEG-4: Context and objectives , 1997, Signal Process. Image Commun..

[11]  Norman I. Badler,et al.  Animating facial expressions , 1981, SIGGRAPH '81.

[12]  Daniel Thalmann,et al.  SMILE: A Multilayered Facial Animation System , 1991, Modeling in Computer Graphics.

[13]  Dimitris N. Metaxas,et al.  The integration of optical flow and deformable models with applications to human face shape and motion estimation , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[14]  Daniel Thalmann,et al.  Complex models for animating synthetic actors , 1991, IEEE Computer Graphics and Applications.

[15]  T. Murakami,et al.  3-D motion estimation of human head for model-based image coding , 1993 .

[16]  Demetri Terzopoulos,et al.  Analysis of facial images using physical and anatomical models , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[17]  John Yen,et al.  Emotionally expressive agents , 1999, Proceedings Computer Animation 1999.

[18]  Keith Waters,et al.  Computer facial animation , 1996 .

[19]  Lance Williams,et al.  Performance-driven facial animation , 1990, SIGGRAPH.

[20]  Daniel Thalmann,et al.  Simulation of Facial Muscle Actions Based on Rational Free Form Deformations , 1992, Comput. Graph. Forum.

[21]  Alex Pentland,et al.  Coding, Analysis, Interpretation, and Recognition of Facial Expressions , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[22]  Keith Waters,et al.  A muscle model for animation three-dimensional facial expression , 1987, SIGGRAPH.

[23]  Pertti Roivainen,et al.  3-D Motion Estimation in Model-Based Facial Image Coding , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[24]  Kiyoharu Aizawa,et al.  Analysis and synthesis of facial image sequences in model-based image coding , 1994, IEEE Trans. Circuits Syst. Video Technol..

[25]  Fabio Lavagetto,et al.  MPEG-4: Audio/video and synthetic graphics/audio for mixed media , 1997, Signal Process. Image Commun..

[26]  Demetri Terzopoulos,et al.  Analysis and Synthesis of Facial Image Sequences Using Physical and Anatomical Models , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[27]  Akikazu Takeuchi,et al.  Speech Dialogue With Facial Displays: Multimodal Human-Computer Conversation , 1994, ACL.

[28]  Thomas S. Huang,et al.  Analysis and compression of facial animation parameter set (FAPs) , 1997, Proceedings of First Signal Processing Society Workshop on Multimedia Signal Processing.

[29]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.