Conversing with the user based on eye-gaze patterns

Motivated by and grounded in observations of eye-gaze patterns in human-human dialogue, this study explores using eye-gaze patterns in managing human-computer dialogue. We developed an interactive system, iTourist, for city trip planning, which encapsulated knowledge of eye-gaze patterns gained from studies of human-human collaboration systems. User study results show that it was possible to sense users' interest based on eye-gaze patterns and manage computer information output accordingly. Study participants could successfully plan their trip with iTourist and positively rated their experience of using it. We demonstrate that eye-gaze could play an important role in managing future multimodal human-computer dialogues.

[1]  Julie C. Sedivy,et al.  Subject Terms: Linguistics Language Eyes & eyesight Cognition & reasoning , 1995 .

[2]  A. Kendon.  Some functions of gaze-direction in social interaction. , 1967, Acta psychologica.

[3]  M K Tanenhaus,et al.  Eye Movements and Lexical Access in Spoken-Language Comprehension: Evaluating a Linking Hypothesis between Fixations and Linguistic Processing , 2000, Journal of psycholinguistic research.

[4]  Roel Vertegaal,et al.  The GAZE groupware system: mediating joint attention in multiparty communication and collaboration , 1999, CHI '99.

[5]  Paul D. Allopenna,et al.  Tracking the Time Course of Spoken Word Recognition Using Eye Movements: Evidence for Continuous Mapping Models , 1998 .

[6]  Kentaro Go,et al.  Overriding errors in a speech and gaze multimodal architecture , 2004, IUI '04.

[7]  Manpreet Kaur,et al.  Where is "it"? Event Synchronization in Gaze-Speech Input Systems , 2003, ICMI '03.

[8]  Shumin Zhai,et al.  What's in the eyes for attentive input , 2003, Commun. ACM.

[9]  Joseph H. Goldberg,et al.  Eye-gaze determination of user intent at the computer interface , 1993 .

[10]  Richard A. Bolt,et al.  A gaze-responsive self-disclosing display , 1990, CHI '90.

[11]  Paul P. Maglio,et al.  SUITOR: an attentive information system , 2000, IUI '00.

[12]  Richard A. Bolt.  Eyes at the interface , 1982, CHI '82.

[13]  Anton Nijholt,et al.  Eye gaze patterns in conversations: there is more to conversational agents than meets the eyes , 2001, CHI.

[14]  Roger M. Cooper,et al.  The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing. , 1974 .

[15]  Päivi Majaranta,et al.  Proactive Response to Eye Movements , 2003, INTERACT.

[16]  B. Velichkovsky.  Communicating attention: Gaze position transfer in cooperative problem solving , 1995 .

[17]  Michael Argyle,et al.  The central Europe experiment: Looking at persons and looking at objects , 1976 .

[18]  Colin Ware,et al.  An evaluation of an eye tracker as a device for computer input2 , 1987, CHI 1987.

[19]  Päivi Majaranta,et al.  Design issues of iDICT: a gaze-assisted translation aid , 2000, ETRA.

[20]  Paul P. Maglio,et al.  A robust algorithm for reading detection , 2001, PUI '01.

[21]  Robert J. K. Jacob,et al.  What you look at is what you get: eye movement-based interaction techniques , 1990, CHI '90.

[22]  C. Phillips,et al.  Journal of Memory and Language , 2001 .

[23]  Pernilla Qvarfordt,et al.  Eyes on multimodal interaction , 2004 .

[24]  Katsumi Tanaka.  A robust selection system using real-time multi-modal user-agent interactions , 1998, IUI '99.

[25]  Dario D. Salvucci.  Inferring intent in eye-based interfaces: tracing eye movements with process models , 1999, CHI '99.

[26]  Zenzi M. Griffin,et al.  PSYCHOLOGICAL SCIENCE Research Article WHAT THE EYES SAY ABOUT SPEAKING , 2022 .

[27]  Sharon L. Oviatt,et al.  Mutual disambiguation of recognition errors in a multimodel architecture , 1999, CHI '99.

[28]  John C. Thomas,et al.  Human factors in computer systems , 1984 .

[29]  Shumin Zhai,et al.  Manual and gaze input cascaded (MAGIC) pointing , 1999, CHI '99.

[30]  C. Ware,et al.  An evaluation of an eye tracker as a device for computer input2 , 1987, CHI '87.

[31]  Ted Selker,et al.  Eye Pattern Analysis in Intelligent Virtual Agents , 2001, IVA.

[32]  Gregory W. Edwards.  A tool for creating eye-aware applications that adapt to changes in user behaviors , 1998, Assets '98.

[33]  John L. Sibert,et al.  The reading assistant: eye gaze triggered auditory prompting for reading remediation , 2000, UIST '00.