Agency in Mid-air Interfaces

Touchless interfaces allow users to view, control and manipulate digital content without physically touching an interface. They are being explored in a wide range of application scenarios from medical surgery to car dashboard controllers. One aspect of touchless interaction that has not been explored to date is the Sense of Agency (SoA). The SoA refers to the subjective experience of voluntary control over actions in the external world. In this paper, we investigated the SoA in touchless systems using the intentional binding paradigm. We first compare touchless systems with physical interactions and then augmented different types of haptic feedback to explore how different outcome modalities influence intentional binding. From our experiments, we demonstrated that an intentional binding effect is observed in both physical and touchless interactions with no statistical difference. Additionally, we found that haptic and auditory feedback help to increase SoA compared with visual feedback in touchless interfaces. We discuss these findings and identify design opportunities that take agency into consideration.

[1]  Mirko Gelsomini,et al.  Designing and evaluating touchless playful interaction for ASD children , 2014, IDC.

[2]  D. Wegner,et al.  Time warp: Authorship shapes the perceived timing of actions and events , 2010, Consciousness and Cognition.

[3]  Patrick Haggard,et al.  Sense of control depends on fluency of action selection, not motor performance , 2012, Cognition.

[4]  Per Ola Kristensson,et al.  I did that! Measuring users' experience of agency in their own actions , 2012, CHI 2012.

[5]  William Del Ra.  Brave NUI world: designing natural user interfaces for touch and gesture by Daniel Wigdor and Dennis Wixon , 2011, SOEN.

[6]  Desney S. Tan,et al.  FingerIO: Using Active Sonar for Fine-Grained Finger Tracking , 2016, CHI.

[7]  Kenton O'Hara,et al.  Touchless interaction in surgery , 2014, CACM.

[8]  Stefan Rank,et al.  Spatial Audio Feedback for Hand Gestures in Games , 2015, SUI.

[9]  Antonio Krüger,et al.  EMPress: Practical Hand Gesture Classification with Wrist-Mounted EMG and Pressure Sensing , 2016, CHI.

[10]  Zhengyou Zhang,et al.  Microsoft Kinect Sensor and Its Effect , 2012, IEEE Multim..

[11]  Diego López-de-Ipiña,et al.  Measuring Software Timing Errors in the Presentation of Visual Stimuli in Cognitive Neuroscience Experiments , 2014, PloS one.

[12]  H. Kashima,et al.  Aberrant sense of agency in patients with schizophrenia: Forward and backward over-attribution of temporal causality during intentional action , 2012, Psychiatry Research.

[13]  Andrew W. Fitzgibbon,et al.  Efficient and precise interactive hand tracking through joint, continuous optimization of pose and correspondences , 2016, ACM Trans. Graph..

[14]  Sriram Subramanian,et al.  UltraHaptics: multi-point mid-air haptic feedback for touch surfaces , 2013, UIST.

[15]  Patrick Haggard,et al.  Dopaminergic medication boosts action–effect binding in Parkinson's disease , 2010, Neuropsychologia.

[16]  M. Voss,et al.  The experience of agency: an interplay between prediction and postdiction , 2013, Front. Psychol..

[17]  Sukhvinder S. Obhi,et al.  Intentional binding and the sense of agency: A review , 2012, Consciousness and Cognition.

[18]  C. S. Mellor,et al.  First Rank Symptoms of Schizophrenia: I. the Frequency in Schizophrenics on Admission to Hospital II. Differences between Individual First Rank Symptoms , 1970, British Journal of Psychiatry.

[19]  Desney S. Tan,et al.  Skinput: appropriating the body as an input surface , 2010, CHI.

[20]  John E. McEneaney.  Agency Effects in Human–Computer Interaction , 2013, Int. J. Hum. Comput. Interact..

[21]  Sukhvinder S. Obhi,et al.  Sense of agency and intentional binding in joint action , 2011, Experimental Brain Research.

[22]  Uwe Handmann,et al.  Contactless Interaction for Automotive Applications , 2013, MuC.

[23]  P. Haggard,et al.  Experience of agency and sense of responsibility , 2011, Consciousness and Cognition.

[24]  Patrick Haggard,et al.  Premotor or Ideomotor: How Does the Experience of Action Come About? , 2013 .

[25]  Michel Beaudouin-Lafon,et al.  Charade: remote control of objects using free-hand gestures , 1993, CACM.

[26]  K. Vogeley,et al.  The “sense of agency” and its underlying cognitive and neural mechanisms , 2008, Consciousness and Cognition.

[27]  Neil Stewart.  Millisecond accuracy video display using OpenGL under Linux , 2006, Behavior research methods.

[28]  Yasutoshi Makino,et al.  HaptoClone (Haptic-Optical Clone) for Mutual Tele-Environment by Real-time 3D Image Transfer with Midair Force Feedback , 2016, CHI.

[29]  M. Mimura,et al.  It’s Not My Fault: Postdictive Modulation of Intentional Binding by Monetary Gains and Losses , 2012, PloS one.

[30]  Patrick Haggard,et al.  Motor activity in the brain precedes our awareness of the intention to move, so how is it that we perceive control? , 2016 .

[31]  P. Haggard,et al.  Voluntary action and conscious awareness , 2002, Nature Neuroscience.

[32]  P. Haggard,et al.  Subliminal priming of actions influences sense of control over effects of action , 2010, Cognition.

[33]  M. Jeannerod.  TO ACT OR NOT TO ACT : PERSPECTIVES ON THE REPRESENTATION OF ACTIONS , 1999 .

[34]  Hiroyuki Shinoda,et al.  HaptoMime: mid-air haptic interaction with a floating virtual screen , 2014, UIST.

[35]  Rachel Nuwer.  Armband adds a twitch to gesture control , 2013 .

[36]  David Coyle,et al.  Empirical Evidence for a Diminished Sense of Agency in Speech Interfaces , 2015, CHI.

[37]  Desney S. Tan,et al.  AirWave: non-contact haptic feedback using air vortex rings , 2013, UbiComp.

[38]  B. Sparrow,et al.  Vicarious agency: experiencing control over the movements of others. , 2004, Journal of personality and social psychology.

[39]  Martina Ziefle,et al.  Design principles of hand gesture interfaces for microinteractions , 2013, DPPI.

[40]  Jörg Müller,et al.  The boomRoom: mid-air direct interaction with virtual sound sources , 2014, CHI.

[41]  P. Haggard,et al.  Who is causing what? The sense of agency is relational and efferent-triggered , 2008, Cognition.

[42]  D. Wegner,et al.  Modulating the sense of agency with external cues , 2009, Consciousness and Cognition.

[43]  Mel Slater,et al.  Body ownership causes illusory self-attribution of speaking and influences subsequent real speaking , 2014, Proceedings of the National Academy of Sciences.

[44]  Stephen A. Brewster,et al.  Do That, There: An Interaction Technique for Addressing In-Air Gesture Systems , 2016, CHI.

[45]  D. Wolpert,et al.  Abnormalities in the awareness of action , 2002, Trends in Cognitive Sciences.

[46]  P. Haggard,et al.  Awareness of action: Inference and prediction , 2008, Consciousness and Cognition.

[47]  B. Libet,et al.  Time of conscious intention to act in relation to onset of cerebral activity (readiness-potential). The unconscious initiation of a freely voluntary act. , 1983 .

[48]  M. Jeannerod,et al.  Beyond Consciousness of External Reality: A “Who” System for Consciousness of Action and Self-Consciousness , 1998, Consciousness and Cognition.

[49]  D. Wegner.  The mind's best trick: how we experience conscious will , 2003, Trends in Cognitive Sciences.

[50]  B. Libet,et al.  Time of conscious intention to act in relation to onset of cerebral activity (readiness-potential). The unconscious initiation of a freely voluntary act. , 1983, Brain : a journal of neurology.

[51]  N. V. van Haren,et al.  Positive priming and intentional binding: Eye-blink rate predicts reward information effects on the sense of agency , 2012, Social neuroscience.

[52]  P. Haggard,et al.  Intentional binding and higher order agency experience , 2010, Consciousness and Cognition.

[53]  Patrick Haggard,et al.  Modulating human sense of agency with non-invasive brain stimulation , 2015, Cortex.

[54]  Sukhvinder S. Obhi,et al.  Sense of agency in joint action: influence of human and computer co-actors , 2011, Experimental Brain Research.

[55]  Ali Israr,et al.  AIREAL: interactive tactile experiences in free air , 2013, ACM Trans. Graph..

[56]  John H. Wright,et al.  Information-Processing Channels in the Tactile Sensory System: A Psychophysical and Physiological Analysis , 2008 .

[57]  Petter Johansson,et al.  Failure to Detect Mismatches Between Intention and Outcome in a Simple Decision Task , 2005, Science.

[58]  A. Sockloff,et al.  Statistical power analysis for the behavioral sciences: (revised edition), by Jacob Cohen. New York: Academic Press, 1977, xv + 474 pp., $24.50. , 1978 .

[59]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[60]  Manos Tsakiris,et al.  The Experience of Agency , 2009 .

[61]  Kasper Hornbæk,et al.  Vulture: a mid-air word-gesture keyboard , 2014, CHI.

[62]  Stefan Rank,et al.  Responsive Environmental Diegetic Audio Feedback for Hand Gestures in Audio-Only Games , 2015, CHI PLAY.

[63]  Mel Slater,et al.  First Person Perspective of Seated Participants Over a Walking Virtual Body Leads to Illusory Agency Over the Walking , 2016, Scientific Reports.

[64]  Ivan Poupyrev,et al.  Soli , 2016, ACM Trans. Graph..

[65]  Daniel Vogel,et al.  Gunslinger: Subtle Arms-down Mid-air Interaction , 2015, UIST.