Referential Gaze Makes a Difference in Spoken Language Comprehension: Human Speaker vs. Virtual Agent Listener Gaze

An interlocutor's referential gaze is of great importance in faceto-face communication as it facilitates spoken language comprehension. People are also able to exploit virtual agent gaze in interactions. Our study addressed effects of human speaker gaze vs. virtual listener gaze on reaction times, accuracy and eye movements. We manipulated: (1) whether the human speaker uttering the sentence was visible, (2) whether the agent listener was present and (3) whether the template following each video matched the scene. Participants saw videos in which a static scene depicting three characters was visible on a screen. We recorded participants ' eye movements as they listened to German SVO sentences describing an interaction between two of these three characters. After each trial a template schematically depicting three characters and their interaction appeared on screen. Participants verified congruence between sentence and template. Participants solved the matching task very well across all conditions. They responded faster to matches than mismatches between sentence and template. Participants were slower when the agent was present. Eye movement results suggest that during the NP2 region participants tended to look at the NP2 referent to a greater extent when the speaker was present compared to the other conditions.