Modeling the Semantic Coordination of Speech and Gesture under Cognitive and Linguistic Constraints

This paper addresses the semantic coordination of speech and gesture, a major prerequisite when endowing virtual agents with convincing multimodal behavior. Previous research has focused on building rule- or data-based models specific for a particular language, culture or individual speaker, but without considering the underlying cognitive processes. We present a flexible cognitive model in which both linguistic as well as cognitive constraints are considered in order to simulate natural semantic coordination across speech and gesture. An implementation of this model is presented and first simulation results, compatible with empirical data from the literature are reported.

[1]  John R Anderson,et al.  An integrated theory of the mind. , 2004, Psychological review.

[2]  Sotaro Kita,et al.  Relations between syntactic encoding and co-speech gestures: Implications for a model of speech and gesture production , 2007 .

[3]  Hao Yan,et al.  Coordination and context-dependence in the generation of embodied conversation , 2000, INLG.

[4]  Janet Beavin Bavelas,et al.  An experimental study of when and how speakers use gestures to communicate , 2002 .

[5]  J. Gregory Trafton,et al.  Linguistic Spatial Gestures , 2010 .

[6]  Stefan Kopp,et al.  Automatic and strategic alignment of co-verbal gestures in dialogue , 2013 .

[7]  G. Bente,et al.  Personalizing e-Learning. The Social Effects of Pedagogical Agents , 2010 .

[8]  Peter Huber,et al.  Generating Culture-Specific Gestures for Virtual Agent Dialogs , 2010, IVA.

[9]  Sotaro Kita,et al.  What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking , 2003 .

[10]  Stefan Kopp,et al.  A spreading-activation model of the semantic coordination of speech and gesture , 2013, CogSci.

[11]  R. Bollet,et al.  Personalizing E-Learning , 2002 .

[12]  David McNeill,et al.  Language and Gesture: Frontmatter , 2000 .

[13]  Stefan Kopp,et al.  GNetIc - Using Bayesian Decision Networks for Iconic Gesture Generation , 2009, IVA.

[14]  Bobby Bodenheimer,et al.  Synthesis and evaluation of linear motion transitions , 2008, TOGS.

[15]  W. Levelt.  Speaking: From Intention to Articulation , 1990 .

[16]  Sotaro Kita,et al.  Competing conceptual representations trigger co-speech representational gestures , 2009 .

[17]  Justine Cassell,et al.  BEAT: the Behavior Expression Animation Toolkit , 2001, Life-like characters.

[18]  Hans-Peter Seidel,et al.  Annotated New Text Engine Animation Animation Lexicon Animation Gesture Profiles MR : . . . JL : . . . Gesture Generation Video Annotated Gesture Script , 2007 .

[19]  Sotaro Kita,et al.  Conceptualisation load triggers gesture production , 2007 .

[20]  Stefan Kopp,et al.  Individualized Gesturing Outperforms Average Gesturing - Evaluating Gesture Production in Virtual Humans , 2010, IVA.

[21]  Asli Ozyurek,et al.  Speech-gesture relationship across languages and in second language learners. Implications for spatial thinking and speaking , 2002 .

[22]  Allan Collins,et al.  A spreading-activation theory of semantic processing , 1975 .

[23]  Susan Duncan,et al.  Growth points in thinking-for-speaking , 1998 .

[24]  Stefan Kopp,et al.  Trading Spaces: How Humans and Humanoids Use Speech and Gesture to Give Directions , 2007 .

[25]  Stefan Kopp,et al.  A Cognitive Model for the Representation and Processing of Shape-Related Gestures , 2003 .

[26]  Stacy Marsella,et al.  Nonverbal Behavior Generator for Embodied Conversational Agents , 2006, IVA.

[27]  Richard J. Gerrig,et al.  Effects of Conversational Pressures on Speech Planning , 2013 .

[28]  Stefan Kopp,et al.  Gestural Alignment in Natural Dialogue , 2012, CogSci.

[29]  Stefan Kopp,et al.  Verbal or Visual? How Information is Distributed across Speech and Gesture in Spatial Dialog , 2006 .

[30]  Martha W. Alibali,et al.  Cognitive skills and gesture–speech redundancy: Formulation difficulty or communicative strategy? , 2011 .

[31]  Stefan Kopp,et al.  A Second Chance to Make a First Impression? How Appearance and Nonverbal Behavior Affect Perceived Warmth and Competence of Virtual Agents over Time , 2012, IVA.

[32]  Martha W. Alibali,et al.  Raise your hand if you’re spatial: Relations between verbal and spatial skills and gesture production , 2007 .