Animated conversation: rule-based generation of facial expression, gesture & spoken intonation for multiple conversational agents

We describe an implemented system which automatically generates and animates conversations between multiple human-like agents with appropriate and synchronized speech, intonation, facial expressions, and hand gestures. Conversation is created by a dialogue planner that produces the text as well as the intonation of the utterances. The speaker/listener relationship, the text, and the intonation in turn drive facial expressions, lip motions, eye gaze, head motion, and arm gestures generators. Coordinated arm, wrist, and hand motions are invoked to create semantically meaningful gestures. Throughout we will use examples from an actual synthesized, fully animated conversation.

[1]  A. Bruce.  Emotional Expression , 1883, The American Naturalist.

[2]  A. Kendon.  Movement coordination in social interaction: some examples described. , 1970, Acta psychologica.

[3]  S. Duncan,et al.  Some Signals and Rules for Taking Speaking Turns in Conversations , 1972 .

[4]  P. Ekman.  Movements with Precise Meanings , 1976 .

[5]  R. Power.  The organisation of purposeful dialogues , 1979 .

[6]  A. Kendon.  Gesticulation and Speech: Two Aspects of the Process of Utterance , 1981 .

[7]  Howard Poizner,et al.  Computer graphic modeling of american sign language , 1983, SIGGRAPH.

[8]  Editors , 1986, Brain Research Bulletin.

[9]  Daniel Thalmann,et al.  Simulation of object and human skin formations in a grasping task , 1989, SIGGRAPH.

[10]  Norman I. Badler,et al.  Making Them Move: Mechanics, Control & Animation of Articulated Figures , 1990 .

[11]  Norman I. Badler,et al.  Strength guided motion , 1990, SIGGRAPH.

[12]  Nadia Magnenat-Thalmann,et al.  Human body deformations using joint-dependent local operators and finite-element theory , 1991 .

[13]  N. Badler,et al.  Linguistic Issues in Facial Animation , 1991 .

[14]  Mark Steedman.  Structure and Intonation , 1991 .

[15]  Michael Girard,et al.  Computer animation of knowledge-based human grasping , 1991, SIGGRAPH.

[16]  Thomas W. Calvert,et al.  Composition of realistic animation sequences for multiple human figures , 1991 .

[17]  D. McNeill,et al.  Gesture and the Poetics of Prose , 1991 .

[18]  Daniel Thalmann,et al.  SMILE: A Multilayered Facial Animation System , 1991, Modeling in Computer Graphics.

[19]  Tosiyasu L. Kunii,et al.  Visual translation: from native language to sign language , 1992, Proceedings IEEE Workshop on Visual Languages.

[20]  E. Prince.  The ZPG Letter: Subjects, Definiteness, and Information-status , 1992 .

[21]  Mark Steedman,et al.  Generating Contextually Appropriate Intonation , 1993, EACL.

[22]  Michael M. Cohen,et al.  Modeling Coarticulation in Synthetic Visual Speech , 1993 .

[23]  Norman I. Badler,et al.  Simulating humans: computer graphics animation and control , 1993 .

[24]  Joseph Rosen,et al.  The virtual sailor: An implementation of interactive human body modeling , 1993, Proceedings of IEEE Virtual Reality Annual International Symposium.

[25]  Akikazu Takeuchi,et al.  Communicative facial displays as a new conversational modality , 1993, INTERCHI.

[26]  K. Tuite.  The production of gesture , 1993 .

[27]  M. Argyle,et al.  Gaze and Mutual Gaze , 1994, British Journal of Psychiatry.

[28]  Matthew Stone,et al.  Modeling the Interaction between Speech and Gesture. , 1994 .

[29]  M. Studdert-Kennedy.  Hand and Mind: What Gestures Reveal About Thought. , 1994 .