Verbal or Visual? How Information is Distributed across Speech and Gesture in Spatial Dialog

In spatial dialog like in direction giving humans make frequent use of speechaccompanying gestures. Some gestures convey largely the same information as speech while others complement speech. This paper reports a study on how speakers distribute meaning across speech and gesture, and depending on what factors. Utterance meaning and the wider dialog context were tested by statistically analyzing a corpus of direction-giving dialogs. Problems of speech production (as indicated by discourse markers and disfluencies), the communicative goals, and the information status were found to be influential, while feedback signals by the addressee do not have any influence.

[1]  Hao Yan,et al.  Paired speech and gesture generation in embodied conversational agents , 2000 .

[2]  Stefan Kopp,et al.  Towards integrated microplanning of language and iconic gesture for multimodal output , 2004, ICMI '04.

[3]  M. Denis.  The description of routes : A cognitive approach to the production of spatial discourse , 1997 .

[4]  James F. Allen,et al.  Draft of DAMSL Dialog Act Markup in Several Layers , 2007 .

[5]  Janet Beavin Bavelas,et al.  An experimental study of when and how speakers use gestures to communicate , 2002 .

[6]  Willem J. M. Levelt,et al.  Gesture and the communicative intention of the speaker , 2005 .

[7]  Heather Shovelton,et al.  When size really matters: How a single semantic feature is represented in the speech and gesture modalities , 2006 .

[8]  S. Goldin-Meadow,et al.  Why people gesture when they speak , 1998, Nature.

[9]  Hao Yan.  Paired Speech and Gesture Generation in Embodied Conversational Agents , 2000 .

[10]  Elizabeth Shriberg,et al.  Phonetic Consequences of Speech Disfluency , 1999 .

[11]  Stefan Kopp.  The spatial specificity of iconic gestures , 2005 .

[12]  Hao Yan,et al.  Coordination and context-dependence in the generation of embodied conversation , 2000, INLG.

[13]  E. Schegloff.  Structures of Social Action: On some gestures' relation to talk , 1985 .

[14]  Michael Kipp,et al.  Gesture generation by imitation: from human behavior to computer character animation , 2005 .

[15]  Sotaro Kita,et al.  What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking , 2003 .

[16]  George Lakoff,et al.  Hedges: A study in meaning criteria and the logic of fuzzy concepts , 1973, J. Philos. Log..

[17]  D. McNeill.  Gesture and Thought , 2005 .

[18]  C. Creider.  Hand and Mind: What Gestures Reveal about Thought , 1994 .

[19]  Ray Jackendoff.  Semantics and Cognition , 1983 .

[20]  Timo Sowa.  Understanding coverbal iconic gestures in shape descriptions , 2006 .