Effects of Ambiguous Gestures and Language on the Time Course of Reference Resolution

Two eye-tracking experiments investigated how and when pointing gestures and location descriptions affect target identification. The experiments investigated the effect of gestures and referring expressions on the time course of fixations to the target, using videos of human gestures and human voice, and animated gestures and synthesized speech. Ambiguous, yet informative pointing gestures elicited attention and facilitated target identification, akin to verbal location descriptions. Moreover, target identification was superior when both pointing gestures and verbal location descriptions were used. These findings suggest that gesture not only operates as a context to verbal descriptions, or that verbal descriptions operate as a context to gesture, but that they complement one another in reference resolution.

[1]  Emiel Krahmer,et al.  Generating Multimodal References , 2007 .

[2]  S. Goldin-Meadow,et al.  The role of gesture in communication and thinking , 1999, Trends in Cognitive Sciences.

[3]  D. Barr.  Analyzing ‘visual world’ eyetracking data using multilevel logistic regression , 2008 .

[4]  Claudio de'Sperati,et al.  Precise Oculomotor Correlates of Visuospatial Mental Rotation and Circular Motion Imagery , 2003, Journal of Cognitive Neuroscience.

[5]  J. Cassell,et al.  Embodied conversational agents , 2000 .

[6]  Rick Dale,et al.  The Self-Organization of Explicit Attitudes , 2009, Psychological science.

[7]  A. Kendon.  Gesture: Visible Action as Utterance , 2004 .

[8]  Spencer D. Kelly,et al.  Gesture Gives a Hand to Language and Learning: Perspectives from Cognitive Neuroscience, Developmental Psychology and Education , 2008, Lang. Linguistics Compass.

[9]  T. Trabasso,et al.  Offering a Hand to Pragmatic Understanding: The Role of Speech and Gesture in Comprehension and Memory , 1999 .

[10]  S. Brennan,et al.  Speakers' eye gaze disambiguates referring expressions early during face-to-face conversation , 2007 .

[11]  Michael K. Tanenhaus,et al.  Embodied communication: Speakers’ gestures affect listeners’ actions , 2009, Cognition.

[12]  G. Miller,et al.  Cognitive science. , 1981, Science.

[13]  M. Tanenhaus,et al.  Approaches to studying world-situated language use : bridging the language-as-product and language-as-action traditions , 2005 .

[14]  L A Thompson,et al.  Children's integration of speech and pointing gestures in comprehension. , 1994, Journal of experimental child psychology.

[15]  Alan Kingstone,et al.  Attentional effects of counterpredictive gaze and arrow cues. , 2004, Journal of experimental psychology. Human perception and performance.

[16]  Claire O'Malley,et al.  Actions speak no louder than words: Symmetrical cross-modal interference effects in the processing of verbal and gestural information. , 1996 .

[17]  M. Alibali,et al.  Effects of Visibility between Speaker and Listener on Gesture Production: Some Gestures Are Meant to Be Seen , 2001 .

[18]  Werner Deutsch,et al.  The Development of Verbal and Nonverbal Devices for Reference. , 1982 .

[19]  E. Maris,et al.  Two Sides of the Same Coin , 2010, Psychological science.

[20]  A. Graesser,et al.  Embodied conversational agents as conversational partners , 2009 .

[21]  H. H. Clark,et al.  Referring as a collaborative process , 1986, Cognition.

[22]  Kara D. Federmeier,et al.  What's 'Right' in Language Comprehension: Event-Related Potentials Reveal Right Hemisphere Language Capabilities , 2008, Lang. Linguistics Compass.

[23]  Michael K. Tanenhaus,et al.  Pragmatic effects on reference resolution in a collaborative task: evidence from eye movements , 2004, Cogn. Sci..

[24]  Daniel C. Richardson,et al.  The Art of Conversation Is Coordination , 2007, Psychological science.

[25]  Margaret Gwendoline Riseborough,et al.  Physiographic gestures as decoding facilitators: Three experiments exploring a neglected facet of communication , 1981 .

[26]  C. Creider.  Hand and Mind: What Gestures Reveal about Thought , 1994 .

[27]  Julie C. Sedivy,et al.  Eye movements and spoken language comprehension: Effects of visual context on syntactic ambiguity resolution , 2002, Cognitive Psychology.

[28]  Kenneth Holmqvist,et al.  What speakers do and what addressees look at: visual attention to gestures in human interaction live and on video , 2006 .

[29]  Robbert-Jan Beun,et al.  Object reference in a shared domain of conversation , 1998 .

[30]  Philip R. Cohen,et al.  Referring as a Collaborative Process , 2003 .

[31]  Stefan Kopp,et al.  Trading Spaces: How Humans and Humanoids Use Speech and Gesture to Give Directions , 2007 .

[32]  D R Olson,et al.  Language and thought: aspects of a cognitive theory of semantics. , 1970, Psychological review.

[33]  H. H. Clark.  Pointing and placing. , 2003 .

[34]  R. Baayen,et al.  Mixed-effects modeling with crossed random effects for subjects and items , 2008 .

[35]  De Ruiter,et al.  Postcards from the mind: The relationship between speech, imagistic gesture and thought , 2007 .

[36]  Ramon C. Littell,et al.  SAS for Linear Models , 2002 .

[37]  A. Bangerter,et al.  Using Pointing and Describing to Achieve Joint Focus of Attention in Dialogue , 2004, Psychological science.