The communicative import of gestures: evidence from a comparative analysis of human-human and human-machine interactions

Does gesturing primarily serve speaker internal purposes, or does it mostly facilitate communication, for example by conveying semantic content, or easing social interaction? To address this question, we asked native speakers of Dutch to retell an animated cartoon to a presumed audiovisual summarizer, a presumed addressee in another room (through web cam), or an addressee in the same room, who could either see them and be seen by them or not. We found that participants produced the least number of gestures when talking to the presumed summarizer. In addition, they produced a smaller proportion of large gestures and almost no pointing gestures. Two perception experiments revealed that observers are sensitive to this difference in gesturing. We conclude that gesture production is not a fully automated speech facilitation process, and that it can convey information about the communicative setting a speaker is in.

[1]  Heather Shovelton,et al.  An experimental investigation of some properties of individual iconic gestures that mediate their communicative power. , 2002, British journal of psychology.

[2]  Alfons Maes,et al.  Referential collaboration with computers. Do we treat computer addressees like humans , 1991 .

[3]  Janet Beavin Bavelas,et al.  Gesturing on the telephone: Independent effects of dialogue and visibility. , 2008 .

[4]  R. Krauss.  Why Do We Gesture When We Speak? , 1998 .

[5]  De Ruiter,et al.  Can gesticulation help aphasic people speak, or rather, communicate? , 2006 .

[6]  William D. Hopkins,et al.  The effect of thought structure on the production of lexical movements , 2002, Brain and Language.

[7]  Robert M. Krauss,et al.  Gesture and Speech in Spontaneous and Rehearsed Narratives , 1994 .

[8]  M. Studdert-Kennedy.  Hand and Mind: What Gestures Reveal About Thought. , 1994 .

[9]  Juergen Luettin,et al.  Audio-Visual Speech Modeling for Continuous Speech Recognition , 2000, IEEE Trans. Multim..

[10]  Samy Bengio,et al.  Automatic analysis of multimodal group actions in meetings , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Willem J. M. Levelt,et al.  Gesture and the communicative intention of the speaker , 2005 .

[12]  M. Swerts,et al.  The Effects of Visual Beats on Prosodic Prominence: Acoustic Analyses, Auditory Perception and Visual Perception. , 2007 .

[13]  A. Kendon.  Do Gestures Communicate? A Review , 1994 .

[14]  M. Alibali,et al.  Effects of Visibility between Speaker and Listener on Gesture Production: Some Gestures Are Meant to Be Seen , 2001 .

[15]  A. Garnham,et al.  The role of conversational hand gestures in a narrative task , 2007 .

[16]  C.D. Martin,et al.  The Media Equation: How People Treat Computers, Television and New Media Like Real People and Places [Book Review] , 1997, IEEE Spectrum.

[17]  Steve Whittaker,et al.  Theories and methods in mediated communication , 2003 .

[18]  Rineke Verbrugge,et al.  Reflective cognition as a secondary task , 2005 .

[19]  E. Aharoni,et al.  Social reactions toward people vs. computers: How mere lables shape interactions , 2007, Comput. Hum. Behav..

[20]  Sotaro Kita,et al.  How representational gestures help speaking , 2000 .

[21]  De Ruiter,et al.  Gesture and speech production , 1998 .

[22]  Akiba A. Cohen,et al.  The Communicative Functions of Hand I1lustrators , 1977 .

[23]  S. Goldin-Meadow,et al.  The role of gesture in communication and thinking , 1999, Trends in Cognitive Sciences.

[24]  Eric Chevalley,et al.  Pointing and Describing in Referential Communication , 2005 .

[25]  Uri Hadar,et al.  Two Types of Gesture and Their Role in Speech Production , 1989 .

[26]  Heather Shovelton,et al.  Mapping the Range of Information Contained in the Iconic Hand Gestures that Accompany Spontaneous Speech , 1999 .

[27]  A. Kendon.  Gesture: Visible Action as Utterance , 2004 .

[28]  J. I.  The Design of Experiments , 1936, Nature.

[29]  Asli Ozyurek.  Do Speakers Design Their Cospeech Gestures for Their Addressees? The Effects of Addressee Location on Representational Gestures , 2002 .

[30]  J. D. Ruiter,et al.  Primary and secondary pragmatic functions of pointing gestures , 2007 .

[31]  D. Barr,et al.  Limits on theory of mind use in adults , 2003, Cognition.

[32]  A. Cohen,et al.  Intentionality in the use of hand illustrators in face-to-face communication situations. , 1973 .

[33]  Clifford Nass,et al.  The media equation - how people treat computers, television, and new media like real people and places , 1996 .

[34]  Susan M. Wagner,et al.  Explaining Math: Gesturing Lightens the Load , 2001, Psychological science.

[35]  Susan Goldin-Meadow,et al.  Gestures convey substantive information about a child’s thoughts to ordinary listeners , 1999 .

[36]  Martha W. Alibali,et al.  Does Sitting on Your Hands Make You Bite Your Tongue? The Effects of Gesture Prohibition on Speech During Motor Descriptions , 2007 .