Accuracy of interpreting pointing gestures in egocentric view

Communicating spatial information by pointing is ubiquitous in human interactions. With the growing use of head-mounted cameras for collaborative purposes, it is important to assess how accurately viewers of the resulting egocentric videos can interpret pointing acts. We conducted an experiment to compare the accuracy of interpreting four different pointing techniques: hand pointing, head pointing, gaze pointing and hand+gaze pointing. Our results suggest that superimposing the gaze information on the egocentric video can enable viewers to determine pointing targets more accurately and more confidently. Hand pointing performed best when the pointing target was straight ahead and head pointing was the least preferred in terms of ease of interpretation. Our results can inform the design of collaborative applications that make use of the egocentric view.

[1]  Richard A. Bolt,et al.  “Put-that-there”: Voice and gesture at the graphics interface , 1980, SIGGRAPH '80.

[2]  Carman Neustaedter,et al.  Exploring video streaming in public settings: shared geocaching over distance using mobile video chat , 2014, CHI.

[3]  Andreas Bulling,et al.  Pupil: an open source platform for pervasive eye tracking and mobile gaze-based interaction , 2014, UbiComp Adjunct.

[4]  Poika Isokoski,et al.  Gaze Augmentation in Egocentric Video Improves Awareness of Intention , 2016, CHI.

[5]  Anthony Dunnigan,et al.  Exploring gestural interaction in smart spaces using head mounted devices with ego-centric sensing , 2014, SUI.

[6]  Robert E. Kraut,et al.  Effects of head-mounted and scene-oriented video systems on remote collaboration on physical tasks , 2003, CHI '03.

[7]  Carman Neustaedter,et al.  Mechanics of Camera Work in Mobile Video Collaboration , 2015, CHI.

[8]  Jun Rekimoto,et al.  JackIn: integrating first-person view with out-of-body vision generation for human-human augmentation , 2014, AH.

[9]  Pat Dugard.  Randomization tests: A new gold standard? , 2014 .

[10]  Aarlenne Z. Khan,et al.  Coordinating one hand with two eyes: optimizing for field of view in a pointing task , 2003, Vision Research.

[11]  Niels Henze,et al.  Modeling Distant Pointing for Compensating Systematic Displacements , 2015, CHI.

[12]  Christopher A. Dickinson,et al.  Coordinating spatial referencing using shared gaze , 2010, Psychonomic bulletin & review.

[13]  Cédric Fleury,et al.  Accuracy of Deictic Gestures to Support Telepresence on Wall-sized Displays , 2015, CHI.

[14]  Vivek K. Goyal,et al.  Mime: compact, low power 3D gesture sensing for interaction with head mounted displays , 2013, UIST.

[15]  J D Crawford,et al.  Role of eye, head, and shoulder geometry in the planning of accurate arm movements. , 2002, Journal of neurophysiology.

[16]  Ying-Chao Tung,et al.  User-Defined Game Input for Smart Glasses in Public Space , 2015, CHI.

[17]  M. Jeannerod,et al.  The contribution of coordinated eye and head movements in hand pointing accuracy , 2004, Experimental Brain Research.

[18]  David P. Carey.  PII: S0960-9822(01)00496-1 , 2001 .

[19]  Peter Fröhlich,et al.  KIBITZER: a wearable system for eye-gaze-based mobile urban exploration , 2010, AH.

[20]  Poika Isokoski,et al.  GazeTorch: Enabling Gaze Awareness in Collaborative Physical Tasks , 2016, CHI Extended Abstracts.

[21]  Marcus Nilsson,et al.  Experiences of using wearable computers for ambient telepresence and remote interaction , 2004, ETP '04.

[22]  Zenzi M. Griffin,et al.  PSYCHOLOGICAL SCIENCE Research Article WHAT THE EYES SAY ABOUT SPEAKING , 2022 .

[23]  Blair MacIntyre,et al.  Browsing the Real-World Wide Web: Maintaining Awareness of Virtual Information in an AR Information Space , 2003, Int. J. Hum. Comput. Interact..

[24]  Andreas Butz,et al.  Free-hand pointing for identification and interaction with distant objects , 2013, AutomotiveUI.

[25]  S. Holm.  A Simple Sequentially Rejective Multiple Test Procedure , 1979 .

[26]  Jun Rekimoto,et al.  LiveSphere: Sharing the Surrounding Visual Environment for Immersive Experience in Remote Collaboration , 2015, Tangible and Embedded Interaction.

[27]  Khan,et al.  PII: S0042-6989(01)00079-7 , 2001 .

[28]  T. Uemura,et al.  Eye-head coordination during lateral gaze in normal subjects. , 1980, Acta oto-laryngologica.

[29]  Päivi Majaranta,et al.  Twenty years of eye typing: systems and design issues , 2002, ETRA.

[30]  Boris M. Velichkovsky,et al.  Gaze transfer in remote cooperation: Is it always helpful to see what your partner is attending to? , 2013, Quarterly journal of experimental psychology.

[31]  W. Miles.  Ocular dominance in human adults. , 1930 .

[32]  Tao Yang,et al.  Eye-Wearable Technology for Machine Maintenance: Effects of Display Position and Hands-free Operation , 2015, CHI.

[33]  Roope Raisamo,et al.  TraQuMe: a tool for measuring the gaze tracking quality , 2014, ETRA.

[34]  Carl Gutwin,et al.  Where are you pointing?: the accuracy of deictic pointing in CVEs , 2010, CHI.

[35]  Bruce H. Thomas,et al.  First Person Indoor/Outdoor Augmented Reality Application: ARQuake , 2002, Personal and Ubiquitous Computing.

[36]  Bilge Mutlu,et al.  Handheld or Handsfree?: Remote Collaboration via Lightweight Head-Mounted Displays and Handheld Devices , 2015, CSCW.

[37]  Manfred Tscheligi,et al.  Shared Gaze in the Car: Towards a Better Driver-Passenger Collaboration , 2014, AutomotiveUI.

[38]  Jie Zhu,et al.  Head orientation and gaze direction in meetings , 2002, CHI Extended Abstracts.