Communication over the Internet using a 3D agent with real-time facial expression analysis, synthesis and text to speech capabilities

We present a system for Internet communication that enhances traditional text-based chatting with real-time analysis and synthesis of the chat parties' facial expressions. It is composed of three main modules: a real-time facial expression analysis component, a 3D agent with facial expression synthesis and text-to-speech capabilities - a talking head, and a communication module. So far we have realized a prototype to find out attractive ways of communication on the Internet and are currently experimenting on how to utilize this new type of modalities in chat communication.

[1]  P. Ekman.  An argument for basic emotions , 1992 .

[2]  W. Lewis Johnson,et al.  STEVE: A Pedagogical Agent for Virtual Reality. , 1998 .

[3]  Kazuo Tanie,et al.  Facial expression communication with FES , 1998, Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No.98EX170).

[4]  Chris Joslin,et al.  Personalized face and speech communication over the Internet , 2001, Proceedings IEEE Virtual Reality 2001.

[5]  A. Murat Tekalp,et al.  Face and 2-D mesh animation in MPEG-4 , 2000, Signal Process. Image Commun..

[6]  Kiyoharu Aizawa,et al.  Analysis and synthesis of facial image sequences in model-based image coding , 1994, IEEE Trans. Circuits Syst. Video Technol..

[7]  Alex Pentland,et al.  Coding, Analysis, Interpretation, and Recognition of Facial Expressions , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  N. P. Chanrasiri.  Real Time Facial Expression Recognition System with Applications to Facial Animation in MPEG-4 , 2001 .

[9]  Thomas S. Huang,et al.  iFACE: A 3D Synthetic Talking Face , 2001, Int. J. Image Graph..

[10]  Keith Waters,et al.  Computer facial animation , 1996 .

[11]  M. Yachida,et al.  Facial expression recognition and its degree estimation , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[12]  Mitsuru Ishizuka,et al.  MAKING THE WEB EMOTIONAL: AUTHORING MULTIMODAL PRESENTATIONS USING A SYNTHETIC 3D AGENT , 2001 .

[13]  Ken Perlin,et al.  Improv: a system for scripting interactive actors in virtual worlds , 1996, SIGGRAPH.

[14]  W. Lewis Johnson,et al.  STEVE (video session): a pedagogical agent for virtual reality , 1998, AGENTS '98.

[15]  Mitsuru Ishizuka,et al.  A 3D Agent with Synthetic Face and Semiautonomous Behavior for Multimodal Presentations , 2001 .

[16]  Mark Steedman,et al.  Generating Facial Expressions for Speech , 1996, Cogn. Sci..

[17]  Hideyuki Nakanishi,et al.  FreeWalk: A 3D Virtual Space for Casual Meetings , 1999, IEEE Multim..

[18]  Mark Steedman,et al.  Animated conversation: rule-based generation of facial expression, gesture & spoken intonation for multiple conversational agents , 1994, SIGGRAPH.