The influence of speaker gaze on listener comprehension: Contrasting visual versus intentional accounts

Previous research has shown that listeners follow speaker gaze to mentioned objects in a shared environment to ground referring expressions, both for human and robot speakers. What is less clear is whether the benefit of speaker gaze is due to the inference of referential intentions (Staudte and Crocker, 2011) or simply the (reflexive) shifts in visual attention. That is, is gaze special in how it affects simultaneous utterance comprehension? In four eye-tracking studies we directly contrast speech-aligned speaker gaze of a virtual agent with a non-gaze visual cue (arrow). Our findings show that both cues similarly direct listeners' attention and that listeners can benefit in utterance comprehension from both cues. Only when they are similarly precise, however, does this equality extend to incongruent cueing sequences: that is, even when the cue sequence does not match the concurrent sequence of spoken referents can listeners benefit from gaze as well as arrows. The results suggest that listeners are able to learn a counter-predictive mapping of both cues to the sequence of referents. Thus, gaze and arrows can in principle be applied with equal flexibility and efficiency during language comprehension.

[1]  A. Kingstone,et al.  The eyes have it! Reflexive orienting is triggered by nonpredictive gaze , 1998 .

[2]  Alan Kingstone,et al.  Attentional control and reflexive orienting to gaze and arrow cues , 2007, Psychonomic bulletin & review.

[3]  A. Karmiloff-Smith,et al.  Are children with autism blind to the mentalistic significance of the eyes , 1995 .

[4]  D. Muir,et al.  Gaze-following : its development and significance , 2007 .

[5]  A. Kingstone,et al.  Are eyes special? It depends on how you look at it , 2002, Psychonomic bulletin & review.

[6]  W. Levelt,et al.  Viewing and naming objects: eye movements during noun phrase production , 1998, Cognition.

[7]  G. Altmann.  Language can mediate eye movement control within 100 milliseconds, regardless of whether there is anything to move the eyes to , 2011, Acta psychologica.

[8]  Umberto Castiello,et al.  How the gaze of others influences object processing , 2008, Trends in Cognitive Sciences.

[9]  Marc Schröder,et al.  The German Text-to-Speech Synthesis System MARY: A Tool for Research, Development and Teaching , 2003, Int. J. Speech Technol..

[10]  M. Tomasello,et al.  Shared intentionality. , 2007, Developmental science.

[11]  Zenzi M. Griffin,et al.  Gaze durations during speech reflect word selection and phonological encoding , 2001, Cognition.

[12]  Alan Kingstone,et al.  Attentional effects of counterpredictive gaze and arrow cues. , 2004, Journal of experimental psychology. Human perception and performance.

[13]  N. Emery,et al.  The eyes have it: the neuroethology, function and evolution of social gaze , 2000, Neuroscience & Biobehavioral Reviews.

[14]  Rajesh P. N. Rao,et al.  "Social" robots are psychological agents for infants: A test of gaze following , 2010, Neural Networks.

[15]  S. Baron-Cohen,et al.  Gaze Perception Triggers Reflexive Visuospatial Orienting , 1999 .

[16]  Matthew W. Crocker,et al.  When Robot Gaze Helps Human Listeners: Attentional versus Intentional Account , 2010 .

[17]  R. Baayen,et al.  Mixed-effects modeling with crossed random effects for subjects and items , 2008 .

[18]  V. Bruce,et al.  Reflexive visual orienting in response to the social attention of others , 1999 .

[19]  S. Brennan,et al.  Speakers' eye gaze disambiguates referring expressions early during face-to-face conversation , 2007 .

[20]  M. Crocker,et al.  Investigating joint attention mechanisms through spoken human–robot interaction , 2011, Cognition.

[21]  John C Trueswell,et al.  Use of Speaker's Gaze and Syntax in Verb Learning , 2009, Language learning and development : the official journal of the Society for Language Development.

[22]  M. Posner,et al.  Orienting of Attention* , 1980, The Quarterly journal of experimental psychology.

[23]  Alexis Héloir,et al.  EMBR: A realtime animation engine for interactive embodied agents , 2009, ACII.

[24]  Andrew P Bayliss,et al.  Gaze and arrow cueing of attention reveals individual differences along the autism spectrum as a function of target context. , 2005, British journal of psychology.

[25]  Jason Tipples,et al.  Orienting to counterpredictive gaze and arrow cues , 2008, Perception & psychophysics.

[26]  C. Vaidya,et al.  Controlling attention to gaze and arrows in childhood: an fMRI study of typical development and Autism Spectrum Disorders. , 2011, Developmental science.

[27]  T. Jaeger,et al.  Categorical Data Analysis: Away from ANOVAs (transformation or not) and towards Logit Mixed Models. , 2008, Journal of memory and language.