MODELING THE PRODUCTION OF COVERBAL ICONIC GESTURES BY LEARNING BAYESIAN DECISION NETWORKS

Expressing spatial information with iconic gestures is abundant in human communication and requires transforming information about a referent into resembling gestural form. This transformation is barely understood and hard to model for expressive virtual agents because it is influenced by the visuospatial features of the referent and the overall discourse context or concomitant speech and its outcome varies considerably across different speakers. We use Bayesian decision networks (BDN) to achieve such a model. Different machine learning techniques are applied to a data corpus of speech and gesture use in a spatial domain to investigate how to learn such networks. Modeling results from an implemented generation system are presented and evaluated against the original corpus data to find out how BDNs can be applied to human gesture formation and which structure learning algorithm performs best.

[1]  Demetri Terzopoulos,et al.  A decision network framework for the behavioral animation of virtual humans , 2007, SCA '07.

[2]  Kevin Murphy,et al.  Bayes net toolbox for Matlab , 1999 .

[3]  Ipke Wachsmuth,et al.  A model for the representation and processing of shape in coverbal iconic gestures , 2005 .

[4]  Cornelia Müller,et al.  Redebegleitende Gesten : Kulturgeschichte, Theorie, Sprachvergleich , 1998 .

[5]  D. McNeill.  Gesture and Thought , 2005 .

[6]  Stefan Kopp,et al.  Gesture in embodied communication and human-computer interaction : 8th International Gesture Workshop, GW 2009, Bielefeld, Germany, February 25-27, 2009 : revised selected papers , 2010 .

[7]  Maurizio Mancini,et al.  Implementing Expressive Gesture Synthesis for Embodied Conversational Agents , 2005, Gesture Workshop.

[8]  P. Spirtes,et al.  An Algorithm for Fast Recovery of Sparse Causal Graphs , 1991 .

[9]  Gregory F. Cooper,et al.  A Bayesian method for the induction of probabilistic networks from data , 1992, Machine Learning.

[10]  Stefan Kopp,et al.  Trading Spaces: How Humans and Humanoids Use Speech and Gesture to Give Directions , 2007 .

[11]  A. Kendon.  Gesture: Visible Action as Utterance , 2004 .

[12]  Martha W. Alibali,et al.  Raise your hand if you’re spatial: Relations between verbal and spatial skills and gesture production , 2007 .

[13]  Hans-Peter Seidel,et al.  Annotated New Text Engine Animation Animation Lexicon Animation Gesture Profiles MR : . . . JL : . . . Gesture Generation Video Annotated Gesture Script , 2007 .

[14]  Stefan Kopp,et al.  GNetIc - Using Bayesian Decision Networks for Iconic Gesture Generation , 2009, IVA.

[15]  Stefan Kopp,et al.  Synthesizing multimodal utterances for conversational agents , 2004, Comput. Animat. Virtual Worlds.

[16]  Zsófia Ruttkay,et al.  Presenting in Style by Virtual Humans , 2007, COST 2102 Workshop.

[17]  Robert Dale,et al.  Referring Expression Generation through Attribute-Based Heuristics , 2009, ENLG.

[18]  Hannes Rieser,et al.  On Factoring Out a Gesture Typology from the Bielefeld Speech-and-Gesture-Alignment Corpus (SAGA) , 2009, Gesture Workshop.

[19]  De Ruiter,et al.  Postcards from the mind: The relationship between speech, imagistic gesture and thought , 2007 .

[20]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[21]  MATT HUENERFAUTH.  Spatial, Temporal, and Semantic Models for American Sign Language Generation: Implications for Gesture Generation , 2008, Int. J. Semantic Comput..

[22]  Stefan Kopp,et al.  Increasing the expressiveness of virtual agents: autonomous generation of speech and gesture for spatial description tasks , 2009, AAMAS.

[23]  Hao Yan,et al.  Coordination and context-dependence in the generation of embodied conversation , 2000, INLG.

[24]  Louis-Philippe Morency,et al.  A probabilistic multimodal approach for predicting listener backchannels , 2009, Autonomous Agents and Multi-Agent Systems.

[25]  Anders L. Madsen,et al.  Hugin - The Tool for Bayesian Networks and Influence Diagrams , 2002, Probabilistic Graphical Models.

[26]  Matthew Stone,et al.  Speaking with hands: creating animated conversational characters from recordings of human performance , 2004, ACM Trans. Graph..

[27]  Jürgen Streeck,et al.  Depicting by gesture , 2008 .

[28]  Stefan Kopp,et al.  Towards integrated microplanning of language and iconic gesture for multimodal output , 2004, ICMI '04.

[29]  Janet Beavin Bavelas,et al.  Gesturing on the telephone: Independent effects of dialogue and visibility. , 2008 .

[30]  Stefan Kopp,et al.  Systematicity and Idiosyncrasy in Iconic Gesture Use: Empirical Analysis and Computational Modeling , 2009, Gesture Workshop.

[31]  Anders L. Madsen,et al.  The Hugin Tool for Probabilistic Graphical Models , 2005, Int. J. Artif. Intell. Tools.

[32]  J. York,et al.  Bayesian Graphical Models for Discrete Data , 1995 .

[33]  Irene Kimbara.  On gestural mimicry , 2006 .

[34]  Ronald A. Howard,et al.  Influence Diagrams , 2005, Decis. Anal..

[35]  S. Lauritzen.  The EM algorithm for graphical association models with missing data , 1995 .

[36]  Tomi Silander,et al.  Comparing Predictive Inference Methods for Discrete , 1997 .