Using Real-Time Facial Expression Analysis and Synthesis

H ow can technology support and invent new ways for geographically separate people to communicate and share experiences? The telephone is one of the most primeval methods, and is still widely used. As the Internet has evolved, however, novel communication methods have emerged. Among them are video tele-phony and videoconferencing technologies, including systems such as FreeWalk, 1 which supports small, casual group meetings in a virtual space with real-time video and audio of the participants. Far more popular, however, are real-time chat applications such as ICQ and MSN messenger, which let people exchange text-or voice-based messages. Among the reasons that chat applications are more popular than real-time video applications is that many people are reluctant to show their faces to their communication partners, particularly in real time. People also tend to communicate more freely when they can hide their identity in informal communications. However, as the use of emoticons suggests, communication without nonverbal information such as facial expressions can be monotonous. To address this, we have developed a system 2 that animates 3D facial agents based on real-time facial expression analysis techniques 3 and research on synthesizing facial expressions and text-to-speech capabilities. 4 (The " Related Work " sidebar discusses this research in more detail.) Our system combines visual, auditory, and primary interfaces to communicate one coherent multimodal chat experience. Users can represent themselves using agents they select from a group that we have predefined. When a user shows a particular expression while typing text, the 3D agent at the receiving end speaks the message aloud while it replays the recognized facial expression sequences and also augments the synthesized voice with appropriate emotional content. Because the visual data exchange is based on the MPEG-4 high-level Facial Animation Parameter for facial expressions (FAP 2), rather than real-time video, our method requires very low bandwidth. The " Web Extras " sidebar offers links to video files of our system at work. Our system Our system consists of three main modules: a real-time facial expression analysis component, which can calculate the MPEG-4 FAP 2; an affec-tive 3D agent with facial expression synthesis and text-to-speech capabilities; and a communication module. We have implemented a prototype to explore attractive Internet communication methods , and have also experimented with using these new modalities in chat communication. How it works As Figure 1 (on p. 22) shows, our system captures the user's current face image using a …