Modeling the Interaction between Speech and Gesture.

Abstract : This paper describes an implemented system that generates spoken dialogue, including speech, intonation, and gesture, using two copies of an identical program that differ only in knowledge of the world and which must cooperate to accomplish a goal. The output of the dialogue generation is used to drive a three-dimensional interactive animated model - two graphic figures on a computer screen who speak and gesture according to the rules of the system. The system is based upon a formal, predictive and explanatory theory of the gesture-speech relationship. A felicitous outcome is a working system to realize autonomous animated conversational agents for virtual reality and other purposes, and a tool for investigating the relationship between speech and gesture.

[1]  Steven K. Feiner,et al.  Automating the generation of coordinated multimedia explanations , 1991, Computer.

[2]  D. Crystal,et al.  Intonation and Grammar in British English , 1967 .

[3]  A. Kendon.  Movement coordination in social interaction: some examples described. , 1970, Acta psychologica.

[4]  Mark Steedman,et al.  Generating Contextually Appropriate Intonation , 1993, EACL.

[5]  Mark Steedman.  Structure and Intonation , 1991 .

[6]  Mark Steedman,et al.  Using context to specify intonation in speech synthesis , 1993, EUROSPEECH.

[7]  E. Prince.  The ZPG Letter: Subjects, Definiteness, and Information-status , 1992 .

[8]  Petr Sgall,et al.  Topic and Focus of a Sentence and the Patterning of a Text , 1988 .

[9]  Candace L. Sidner,et al.  Attention, Intentions, and the Structure of Discourse , 1986, CL.

[10]  N. Badler,et al.  Linguistic Issues in Facial Animation , 1991 .

[11]  Wolfgang Wahlster,et al.  Designing Illustrated Texts: How Language Production Is Influenced by Graphics Generation , 1991, EACL.

[12]  D. McNeill,et al.  Gesture and the Poetics of Prose , 1991 .

[13]  R. Power.  The organisation of purposeful dialogues , 1979 .

[14]  M. Studdert-Kennedy.  Hand and Mind: What Gestures Reveal About Thought. , 1994 .

[15]  A. Kendon.  Do Gestures Communicate? A Review , 1994 .

[16]  Mark Steedman,et al.  Animated conversation: rule-based generation of facial expression, gesture & spoken intonation for multiple conversational agents , 1994, SIGGRAPH.

[17]  K. Tuite.  The production of gesture , 1993 .