Seeing and Being Seen: The Effects on Gesture Production

Speakers are argued to adapt their language production to their addressee’s needs. For instance, speakers produce fewer and smaller hand gesture sw hen interlocutors cannot see each other. Yet is this because speakers know their addressee cannot see them, or because they themselves do not see their addressee? By means of computer-mediated communication we manipulated these factors independently. We found that speakers took into account what their addressee saw. They produced more and larger gestures when they knew the addressee could see them. Seeing the addressee increased gesture production only if speakers could readil yi nterpret the addressee’s eye gaze, which is not usually the case in mediated interaction. Adding this affordance resulted in gesturing being similar in mediated and unmediated communication.

[1]  Robert M. Krauss,et al.  Gesture and Speech in Spontaneous and Rehearsed Narratives , 1994 .

[2]  John C. Tang,et al.  What video can and cannot do for collaboration: A case study , 2005, Multimedia Systems.

[3]  Pierre Feyereisen,et al.  The Meaning of Gestures - What Can Be Understood Without Speech , 1988 .

[4]  S. Kita,et al.  Spontaneous gestures during mental rotation tasks: insights into the microdevelopment of the motor strategy. , 2008, Journal of experimental psychology. General.

[5]  A. Kendon.  Some functions of gaze-direction in social interaction. , 1967, Acta psychologica.

[6]  M. Argyle,et al.  Gaze and Mutual Gaze , 1994, British Journal of Psychiatry.

[7]  Susan Goldin-Meadow,et al.  Gestures convey substantive information about a child’s thoughts to ordinary listeners , 1999 .

[8]  M. Alibali,et al.  Gesture and the process of speech production: We think, therefore we gesture , 2000 .

[9]  V. Ferreira,et al.  Don't Talk About Pink Elephants! , 2006, Psychological science.

[10]  S. Brennan,et al.  Computer-Mediated Communication: Cognitive Science Approach , 2006 .

[11]  A. Avramides.  Studies in the Way of Words , 1992 .

[12]  Susan Brennan,et al.  Why do electronic conversations seem less polite? the costs and benefits of hedging , 1999, WACC '99.

[13]  Herbert H. Clark,et al.  Grounding in communication , 1991, Perspectives on socially shared cognition.

[14]  Janet Beavin Bavelas,et al.  Gesturing on the telephone: Independent effects of dialogue and visibility. , 2008 .

[15]  D. Barr,et al.  Limits on theory of mind use in adults , 2003, Cognition.

[16]  M. Studdert-Kennedy.  Hand and Mind: What Gestures Reveal About Thought. , 1994 .

[17]  D. McNeill,et al.  Speech-gesture mismatches: Evidence for one underlying representation of linguistic and nonlinguistic information , 1998 .

[18]  H. H. Clark,et al.  Using uh and um in spontaneous speaking , 2002, Cognition.

[19]  John Short,et al.  The social psychology of telecommunications , 1976 .

[20]  David W. Park,et al.  Interpersonal Effects in Computer-Mediated Interaction , 1994 .

[21]  P. Ekman,et al.  The Repertoire of Nonverbal Behavior: Categories, Origins, Usage, and Coding , 1969 .

[22]  Christopher A. Dickinson,et al.  Coordinating cognition: The costs and benefits of shared gaze during collaborative search , 2008, Cognition.

[23]  Dicky Gilbers,et al.  Language and Cognition 2 , 1992 .

[24]  W. H. Sumby,et al.  Visual contribution to speech intelligibility in noise , 1954 .

[25]  Eric Chevalley,et al.  Pointing and Describing in Referential Communication , 2005 .

[26]  Susan Goldin-Meadow,et al.  When gesture does and does not promote learning , 2010, Language and Cognition.

[27]  Heather Shovelton,et al.  Mapping the Range of Information Contained in the Iconic Hand Gestures that Accompany Spontaneous Speech , 1999 .

[28]  A. Kendon.  Gesture: Visible Action as Utterance , 2004 .

[29]  A. Cohen,et al.  Intentionality in the use of hand illustrators in face-to-face communication situations. , 1973 .

[30]  D. Over,et al.  Studies in the Way of Words. , 1989 .

[31]  Sotaro Kita,et al.  Conceptualisation load triggers gesture production , 2007 .

[32]  Sotaro Kita,et al.  Attention to Speech-Accompanying Gestures: Eye Movements and Information Uptake , 2009, Journal of nonverbal behavior.

[33]  Akiba A. Cohen,et al.  The Communicative Functions of Hand I1lustrators , 1977 .

[34]  R. Krauss.  Why Do We Gesture When We Speak? , 1998 .

[35]  Uri Hadar,et al.  Two Types of Gesture and Their Role in Speech Production , 1989 .

[36]  Asli Ozyurek.  Do Speakers Design Their Cospeech Gestures for Their Addressees? The Effects of Addressee Location on Representational Gestures , 2002 .

[37]  S. Brennan,et al.  Speakers' eye gaze disambiguates referring expressions early during face-to-face conversation , 2007 .

[38]  Adam Kendon,et al.  How gestures can become like words , 1988 .

[39]  M. Swerts,et al.  The Effects of Visual Beats on Prosodic Prominence: Acoustic Analyses, Auditory Perception and Visual Perception. , 2007 .

[40]  Monica Bucciarelli,et al.  The Deep Versus the Shallow: Effects of Co-Speech Gestures in Learning From Discourse , 2008, Cogn. Sci..

[41]  M. Alibali,et al.  Effects of Visibility between Speaker and Listener on Gesture Production: Some Gestures Are Meant to Be Seen , 2001 .

[42]  J. P. Foley,et al.  Gesture and Environment , 1942 .

[43]  Emiel Krahmer,et al.  The communicative import of gestures: evidence from a comparative analysis of human-human and human-machine interactions , 2007, AVSP.

[44]  A. Garnham,et al.  The role of conversational hand gestures in a narrative task , 2007 .