Eye-trackers and Multimodal Communication Studies

This article provides an overview of eye-tracking technology in multimodal communication studies. It presents a short review of the human visual perception system and the eye-tracking technology, and discusses two types of eye-gaze studies as examples of how eye-trackers can be used in interaction management: in turn-taking analysis and involvement in conversation.

[1]  Keith S. Karn,et al.  Commentary on Section 4. Eye tracking in human-computer interaction and usability research: Ready to deliver the promises. , 2003 .

[2]  M. Argyle,et al.  Gaze and Mutual Gaze , 1994, British Journal of Psychiatry.

[3]  Masafumi Nishida,et al.  Turn-alignment using eye-gaze and speech in conversational interaction , 2010, INTERSPEECH.

[4]  Ronald A. Rensink,et al.  TO SEE OR NOT TO SEE: The Need for Attention to Perceive Changes in Scenes , 1997 .

[5]  Yasuharu Den,et al.  Prosody-based detection of the context of backchannel responses , 1998, ICSLP.

[6]  Mattias Heldner,et al.  Prosodic features of very short utterances in dialogue , 2009 .

[7]  A. Ichikawa,et al.  An Analysis of Turn-Taking and Backchannels Based on Prosodic and Syntactic Features in Japanese Map Task Dialogs , 1998, Language and speech.

[8]  B. Velichkovsky,et al.  On the control of visual fixation durations in free viewing of complex images , 2011, Attention, perception & psychophysics.

[9]  Laura Chamberlain.  Eye Tracking Methodology; Theory and Practice , 2007 .

[10]  Isabella Poggi.  The Lexicon and the Alphabet of Gesture, Gaze, and Touch , 2001, IVA.

[11]  Christof Koch,et al.  Modeling attention to salient proto-objects , 2006, Neural Networks.

[12]  B. Velichkovsky,et al.  Time course of information processing during scene perception: The relationship between saccade amplitude and fixation duration , 2005 .

[13]  Nancy Millette,et al.  How People Look at Pictures , 1935 .

[14]  Justine Cassell,et al.  Making Space for Voice: Technologies to Support Children’s Fantasy and Storytelling , 2001, Personal and Ubiquitous Computing.

[15]  Janet Beavin Bavelas.  7 The Two Solitudes: Reconciling Social Psychology and Language and Social Interaction , 2004 .

[16]  Kenneth Holmqvist,et al.  What speakers do and what addressees look at: visual attention to gestures in human interaction live and on video , 2006 .

[17]  Justine Cassell,et al.  BEAT: the Behavior Expression Animation Toolkit , 2001, Life-like characters.

[18]  Gustav Kuhn,et al.  Visual cognition during real social interaction , 2012, Front. Hum. Neurosci..

[19]  E. B. Huey,et al.  Preliminary experiments in the physiology and psychology of reading , 1898 .

[20]  Mattias Heldner,et al.  Pause and gap length in face-to-face interaction , 2009, INTERSPEECH.

[21]  D. McNeill,et al.  Speech-gesture mismatches: Evidence for one underlying representation of linguistic and nonlinguistic information , 1998 .

[22]  Hao Yan,et al.  More than just a pretty face: conversational protocols and the affordances of embodiment , 2001, Knowl. Based Syst..

[23]  Jens Edlund,et al.  The effects of prosodic features on the interpretation of clarification ellipses , 2005, INTERSPEECH.

[24]  Rudolf Groner,et al.  Attention and eye movement control: An overview , 2005, European archives of psychiatry and neurological sciences.

[25]  A. Siegel,et al.  Keeping the Floor in Multiparty Conversations: Intonation, Syntax, and Pause , 2003 .

[26]  Hirotake Yamazoe,et al.  Gaze-communicative behavior of stuffed-toy robot with joint attention and eye contact based on ambient gaze-tracking , 2007, ICMI '07.

[27]  Kristiina Jokinen,et al.  Visual interaction and conversational activity , 2012, Gaze-In '12.

[28]  A. L. I︠A︡rbus.  Eye Movements and Vision , 1967 .

[29]  M. Knapp,et al.  The Interaction of Visual and Verbal Features in Human Communication , 1992 .

[30]  Päivi Majaranta,et al.  From Gaze Control to Attentive Interfaces , 2005 .

[31]  C. Goodwin.  Conversational Organization: Interaction Between Speakers and Hearers , 1981 .

[32]  Masafumi Nishida,et al.  Eye-gaze experiments for conversation monitoring , 2009, IUCS.

[33]  Z. Pylyshyn.  Is vision continuous with cognition? The case for cognitive impenetrability of visual perception. , 1999, The Behavioral and brain sciences.

[34]  Päivi Majaranta,et al.  CHAPTER 9 – Text Entry by Gaze: Utilizing Eye Tracking , 2007 .

[35]  Kenneth Holmqvist,et al.  Keeping an eye on gestures: Visual perception of gestures in face-to-face communication , 1999 .

[36]  M. Land.  Eye movements and the control of actions in everyday life , 2006, Progress in Retinal and Eye Research.

[37]  A. Kendon.  Some functions of gaze-direction in social interaction. , 1967, Acta psychologica.

[38]  Yukiko I. Nakano,et al.  Non-Verbal Cues for Discourse Structure , 2022 .

[39]  Michael F. McTear,et al.  Spoken Dialogue Systems , 2009, Spoken Dialogue Systems.

[40]  Seiichi Yamamoto,et al.  On eye-gaze and turn-taking , 2010, EGIHMI '10.

[41]  Masafumi Nishida,et al.  Collecting and Annotating Conversational Eye-Gaze Data , 2010 .

[42]  Toyoaki Nishida,et al.  Attentional Behaviors as Nonverbal Communicative Signals in Situated Interactions with Conversational Agents , 2007 .

[43]  Justine Cassell,et al.  Relational agents: a model and implementation of building user trust , 2001, CHI.

[44]  Jens Allwood,et al.  Hesitation in Intercultural Communication: Some Observations and Analyses on Interpreting Shoulder Shrugging , 2010, Culture and Computing.

[45]  Jens Edlund,et al.  Higgins - a spoken dialogue system for investigating error handling techniques , 2004, INTERSPEECH.

[46]  Ronald A. Rensink.  The Dynamic Representation of Scenes , 2000 .

[47]  M. Land.  Vision, eye movements, and natural behavior , 2009, Visual Neuroscience.