Automatic and strategic alignment of co-verbal gestures in dialogue

Spontaneous co-verbal gestures are one means for speakers to align with each other and coordinate their interaction. In this chapter we review current empirical findings of this phenomenon and propose two kinds of partner-specific adaptations of gesturing in dialogue: Automatic alignment that occurs through priming via sensorimotor resonances and direct perception-action-links, while strategic adaptations result from flexible production choices based on recipient design and grounding. Both mechanisms are suggested to work to some degree independently and simultaneously, possibly affecting different features of the same gesture. An integrated process models of speech-gesture production and understanding is proposed that explains these adaptation mechanisms and how they may work together.

[1]  Kimberly J. Montgomery,et al.  Communicative hand gestures and object-directed hand movements activated the mirror neuron system. , 2007, Social cognitive and affective neuroscience.

[2]  Sotaro Kita,et al.  What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking , 2003 .

[3]  Anna K. Kuhlen,et al.  Two Minds, One Dialog , 2010 .

[4]  Stefan Kopp,et al.  Adaptive expressiveness: virtual conversational agents that can align to their interaction partner , 2010, AAMAS.

[5]  Katja U. Likowski.  Facial mimicry, valence evaluation or emotional reaction? : mechanisms underlying the modulation of congruent and incongruent facial reactions to emotional facial expressions , 2011 .

[6]  Stefan Kopp,et al.  Trading Spaces: How Humans and Humanoids Use Speech and Gesture to Give Directions , 2007 .

[7]  Irene Kimbara.  Gesture Form Convergence in Joint Description , 2008 .

[8]  Stefan Kopp,et al.  Regulating Dialogue with Gestures - Towards an Empirically Grounded Simulation with Conversational Agents , 2011, SIGDIAL Conference.

[9]  Stefan Kopp,et al.  Embodied Gesture Processing: Motor-based Perception-Action Integration in Social Artificial Agents , 2011 .

[10]  A. Garnham,et al.  The role of conversational hand gestures in a narrative task , 2007 .

[11]  Karl J. Friston,et al.  Predictive coding: an account of the mirror neuron system , 2007, Cognitive Processing.

[12]  J. D. Ruiter.  The production of gesture and speech , 2000 .

[13]  Fey Parrill,et al.  Seeing and Hearing Double: The Influence of Mimicry in Speech and Gesture on Observers , 2006 .

[14]  T. Chartrand,et al.  Using Nonconscious Behavioral Mimicry to Create Affiliation and Rapport , 2003, Psychological science.

[15]  Günther Knoblich,et al.  Favouritism in the motor system: social interaction modulates action simulation , 2010, Biology Letters.

[16]  Janet Beavin Bavelas,et al.  Gesturing on the telephone: Independent effects of dialogue and visibility. , 2008 .

[17]  Stefan Kopp,et al.  A spreading-activation model of the semantic coordination of speech and gesture , 2013, CogSci.

[18]  Pierre Feyereisen,et al.  Mental Imagery and Production of Hand Gestures While Speaking in Younger and Older Adults , 1999 .

[19]  Stefan Kopp,et al.  Multimodal Communication from Multimodal Thinking - towards an Integrated Model of Speech and Gesture Production , 2008, Int. J. Semantic Comput..

[20]  L. Tickle-Degnen,et al.  The Nature of Rapport and Its Nonverbal Correlates , 1990 .

[21]  Scott T. Grafton,et al.  Action outcomes are represented in human inferior frontoparietal cortex. , 2008, Cerebral cortex.

[22]  J. Holler,et al.  Co-Speech Gesture Mimicry in the Process of Collaborative Referring During Face-to-Face Dialogue , 2011 .

[23]  P. Healey,et al.  Tracking Lexical and Syntactic Alignment in Conversation , 2010 .

[24]  Autumn B. Hostetter,et al.  When do gestures communicate? A meta-analysis. , 2011, Psychological bulletin.

[25]  Susan Goldin-Meadow,et al.  Gesturing Saves Cognitive Resources When Talking About Nonpresent Objects , 2010, Cogn. Sci..

[26]  A. Kendon.  Gesture: Visible Action as Utterance , 2004 .

[27]  Martha W. Alibali,et al.  Raise your hand if you’re spatial: Relations between verbal and spatial skills and gesture production , 2007 .

[28]  Yihsiu Chen,et al.  Language and Gesture: Lexical gestures and lexical access: a process model , 2000 .

[29]  Stefan Kopp,et al.  Data-based analysis of speech and gesture: the Bielefeld Speech and Gesture Alignment corpus (SaGA) and its applications , 2013, Journal on Multimodal User Interfaces.

[30]  H. Wallbott.  Bodily expression of emotion , 1998 .

[31]  Stefan Kopp,et al.  Synthesizing multimodal utterances for conversational agents , 2004, Comput. Animat. Virtual Worlds.

[32]  J. Bavelas,et al.  Gestures Specialized for Dialogue , 1995 .

[33]  Boaz Keysar.  Language users as problem solvers: Just what ambiguity problem do they solve? , 1998 .

[34]  Timothy M. Gann,et al.  Speaking from experience: audience design as expert performance , 2014 .

[35]  Stefan Kopp,et al.  Gestural Alignment in Natural Dialogue , 2012, CogSci.

[36]  M. Pickering,et al.  Toward a mechanistic psychology of dialogue , 2004, Behavioral and Brain Sciences.

[37]  Stefan Kopp,et al.  The Relation of Speech and Gestures: Temporal Synchrony Follows Semantic Synchrony , 2011 .

[38]  Stefan Kopp,et al.  Increasing the expressiveness of virtual agents: autonomous generation of speech and gesture for spatial description tasks , 2009, AAMAS.

[39]  Arthur N. Wiens,et al.  Personality Correlates of Nonverbal Interview Behavior. , 1980 .

[40]  T. Chartrand,et al.  The chameleon effect: the perception-behavior link and social interaction. , 1999, Journal of personality and social psychology.

[41]  Irene Kimbara.  On gestural mimicry , 2006 .

[42]  Autumn B. Hostetter,et al.  Visible embodiment: Gestures as simulated action , 2008, Psychonomic bulletin & review.

[43]  D. McNeill.  Gesture and Thought , 2005 .

[44]  Susan Duncan,et al.  Growth points in thinking-for-speaking , 1998 .

[45]  G. Rizzolatti,et al.  A unifying view of the basis of social cognition , 2004, Trends in Cognitive Sciences.

[46]  R. B. van Baaren.  The Parrot Effect How to Increase Tip Size , 2005 .

[47]  M. Swerts,et al.  Adaptation in Gesture: Converging Hands or Converging Minds?. , 2012 .

[48]  S. Brennan,et al.  Attenuating Information in Spoken Communication: For the Speaker, or for the Addressee?. , 2010 .