Towards Natural Gesture Synthesis: Evaluating Gesture Units in a Data-Driven Approach to Gesture Synthesis

Virtual humans still lack naturalness in their nonverbal behaviour. We present a data-driven solution that moves towards a more natural synthesis of hand and arm gestures by recreating gestural behaviour in the style of a human performer. Our algorithm exploits the concept of gesture units to make the produced gestures a continuous flow of movement. We empirically validated the use of gesture units in the generation and show that it causes the virtual human to be perceived as more natural.

[1]  Radoslaw Niewiadomski,et al.  Multimodal Complex Emotions: Gesture Expressivity and Blended Facial Expressions , 2006, Int. J. Humanoid Robotics.

[2]  Stefan Kopp,et al.  Synthesizing multimodal utterances for conversational agents: Research Articles , 2004 .

[3]  Ipke Wachsmuth,et al.  Gesture and Sign Language in Human-Computer Interaction , 1998, Lecture Notes in Computer Science.

[4]  Matthew Stone,et al.  Speaking with hands: creating animated conversational characters from recordings of human performance , 2004, ACM Trans. Graph..

[5]  Han Noot,et al.  Gesture in Style , 2003, Gesture Workshop.

[6]  Michael Kipp,et al.  Gesture generation by imitation: from human behavior to computer character animation , 2005 .

[7]  D. McNeill.  Gesture and Thought , 2005 .

[8]  C. Creider.  Hand and Mind: What Gestures Reveal about Thought , 1994 .

[9]  B. Hartmann,et al.  Design and Evaluation of Expressive Gesture Synthesis for Embodied Conversational , 2005 .

[10]  J. D. Ruiter.  The production of gesture and speech , 2000 .

[11]  Jessica K. Hodgins,et al.  Motion capture-driven simulations that hit and react , 2002, SCA '02.

[12]  E. Schegloff.  Structures of Social Action: On some gestures' relation to talk , 1985 .

[13]  Maurizio Mancini,et al.  Implementing Expressive Gesture Synthesis for Embodied Conversational Agents , 2005, Gesture Workshop.

[14]  Nicolas Courty,et al.  Gesture in Human-Computer Interaction and Simulation , 2006 .

[15]  Catherine Pelachaud,et al.  From brows to trust: evaluating embodied conversational agents , 2004 .

[16]  Paul Boersma,et al.  Praat: doing phonetics by computer , 2003 .

[17]  A. Kendon.  Gesticulation and Speech: Two Aspects of the Process of Utterance , 1981 .

[18]  Mark Steedman,et al.  Animated conversation: rule-based generation of facial expression, gesture & spoken intonation for multiple conversational agents , 1994, SIGGRAPH.

[19]  Myung-Kwan Park,et al.  The EPP and the Subject Condition under Sluicing , 2003, Linguistic Inquiry.

[20]  Sotaro Kita,et al.  Movement Phase in Signs and Co-Speech Gestures, and Their Transcriptions by Human Coders , 1997, Gesture Workshop.

[21]  Michael Neff,et al.  An annotation scheme for conversational gestures: how to economically capture timing and form , 2007, Lang. Resour. Evaluation.

[22]  Marisa E. Campbell,et al.  SIGGRAPH 2004 , 2004, INTR.

[23]  David C. Brogan,et al.  Animating human athletics , 1995, SIGGRAPH.

[24]  J. Cassell,et al.  Embodied conversational agents , 2000 .

[25]  Mel Slater,et al.  Building Expression into Virtual Characters , 2006, Eurographics.

[26]  Norman I. Badler,et al.  The EMOTE model for effort and shape , 2000, SIGGRAPH.

[27]  Norman I. Badler,et al.  Creating Interactive Virtual Humans: Some Assembly Required , 2002, IEEE Intell. Syst..

[28]  S. Frey.  ¬Die Macht des Bildes : der Einfluß der nonverbalen Kommunikation auf Kultur und Politik , 1999 .

[29]  Justine Cassell,et al.  BEAT: the Behavior Expression Animation Toolkit , 2001, Life-like characters.

[30]  Michael Neff,et al.  Modeling tension and relaxation for computer animation , 2002, SCA '02.

[31]  Mark Steedman,et al.  Information Structure and the Syntax-Phonology Interface , 2000, Linguistic Inquiry.

[32]  Maurizio Mancini,et al.  Design and evaluation of expressive gesture synthesis for embodied conversational agents , 2005, AAMAS '05.

[33]  Stacy Marsella,et al.  Nonverbal Behavior Generator for Embodied Conversational Agents , 2006, IVA.

[34]  F. Thomas,et al.  The illusion of life : Disney animation , 1981 .

[35]  William H. Press,et al.  Numerical recipes in C. The art of scientific computing , 1987 .

[36]  Paul Boersma,et al.  Praat, a system for doing phonetics by computer , 2002 .

[37]  W. Press,et al.  Numerical Recipes in C++: The Art of Scientific Computing (2nd edn)1 Numerical Recipes Example Book (C++) (2nd edn)2 Numerical Recipes Multi-Language Code CD ROM with LINUX or UNIX Single-Screen License Revised Version3 , 2003 .

[38]  J. M. Atkinson.  Structures of Social Action: Contents , 1985 .

[39]  Nicole C. Krämer,et al.  Effects of Embodied Interface Agents and Their Gestural Activity , 2003, IVA.

[40]  A. Kendon.  Gesture: Visible Action as Utterance , 2004 .

[41]  Michael Kipp,et al.  ANVIL - a generic annotation tool for multimodal dialogue , 2001, INTERSPEECH.

[42]  Hans-Peter Seidel,et al.  Annotated New Text Engine Animation Animation Lexicon Animation Gesture Profiles MR : . . . JL : . . . Gesture Generation Video Annotated Gesture Script , 2007 .

[43]  Stefan Kopp,et al.  Synthesizing multimodal utterances for conversational agents , 2004, Comput. Animat. Virtual Worlds.

[44]  Michael Neff,et al.  AER: aesthetic exploration and refinement for expressive character animation , 2005, SCA '05.

[45]  Petros Faloutsos,et al.  The virtual stuntman: dynamic characters with a repertoire of autonomous motor skills , 2001, Comput. Graph..

[46]  C. Nass,et al.  Truth is beauty: researching embodied conversational agents , 2001 .

[47]  Antonio Camurri,et al.  Gesture-Based Communication in Human-Computer Interaction , 2003, Lecture Notes in Computer Science.

[48]  R. McCrae,et al.  An introduction to the five-factor model and its applications. , 1992, Journal of personality.