User Representations in Human-Computer Interaction

Cursors, avatars, virtual hands or tools, and other rendered graphical objects, enable users to interact with computers such as PCs, game consoles or virtual reality systems. We analyze the role of...

[1]  P. Haggard,et al.  Having a body versus moving your body: How agency structures body-ownership , 2006, Consciousness and Cognition.

[2]  Jörg Müller,et al.  LeviCursor: Dexterous Interaction with a Levitating Object , 2018, ISS.

[3]  Atsushi Iriki,et al.  Self-images in the video monitor coded by monkey intraparietal neurons , 2001, Neuroscience Research.

[4]  Austin Rallis,et al.  Getting a handle on virtual tools: An examination of the neuronal activity associated with virtual tool use , 2018, Neuropsychologia.

[5]  James W. Moore,et al.  Revisiting the link between body and agency: visual movement congruency enhances intentional binding but is not body-specific , 2018, Scientific Reports.

[6]  S. Debener,et al.  The Senses of Agency and Ownership: A Review , 2018, Front. Psychol..

[7]  D. Bavelier,et al.  Exercising your brain: a review of human brain plasticity and training-induced learning. , 2008, Psychology and aging.

[8]  Tabitha C. Peck,et al.  A threat to a virtual hand elicits motor cortex activation , 2014, Experimental Brain Research.

[9]  Kasper Hornbæk,et al.  Tool Extension in Human-Computer Interaction , 2019, CHI.

[10]  David Swapp,et al.  Interaction with co-located haptic feedback in virtual reality , 2006, Virtual Reality.

[11]  A. Berti,et al.  When Far Becomes Near: Remapping of Space by Tool Use , 2000, Journal of Cognitive Neuroscience.

[12]  Ivan Poupyrev,et al.  Virtual object manipulation on a table-top AR environment , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[13]  Per Ola Kristensson,et al.  I did that! Measuring users' experience of agency in their own actions , 2012, CHI 2012.

[14]  Renaud Blanch,et al.  Semantic pointing: improving target acquisition with control-display ratio adaptation , 2004, CHI.

[15]  Maria V. Sanchez-Vives,et al.  Extending Body Space in Immersive Virtual Reality: A Very Long Arm Illusion , 2012, PloS one.

[16]  Tomohisa Asai.  Feedback control of one’s own action: Self-other sensory attribution in motor control , 2015, Consciousness and Cognition.

[17]  D. Burr,et al.  Combining visual and auditory information. , 2006, Progress in brain research.

[18]  Mel Slater,et al.  Drumming in Immersive Virtual Reality: The Body Shapes the Way We Play , 2013, IEEE Transactions on Visualization and Computer Graphics.

[19]  Ferran Argelaguet,et al.  The role of interaction in virtual embodiment: Effects of the virtual hand representation , 2016, 2016 IEEE Virtual Reality (VR).

[20]  M. Slater,et al.  The building blocks of the full body ownership illusion , 2013, Front. Hum. Neurosci..

[21]  B. Sparrow,et al.  Vicarious agency: experiencing control over the movements of others. , 2004, Journal of personality and social psychology.

[22]  Orestis Georgiou,et al.  Touchless Haptic Feedback for VR Rhythm Games , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[23]  Martin Molina,et al.  Natural user interfaces for human-drone multi-modal interaction , 2016, 2016 International Conference on Unmanned Aircraft Systems (ICUAS).

[24]  Maria V. Sanchez-Vives,et al.  Virtual Hand Illusion Induced by Visuomotor Correlations , 2010, PloS one.

[25]  Antonella Maselli.  Allocentric and egocentric manipulations of the sense of self-location in full-body illusions and their relation with the sense of body ownership , 2015, Cognitive Processing.

[26]  Mel Slater,et al.  Beaming into the News: A System for and Case Study of Tele-Immersive Journalism , 2018, IEEE Computer Graphics and Applications.

[27]  Ravin Balakrishnan,et al.  "Beating" Fitts' law: virtual enhancements for pointing facilitation , 2004, Int. J. Hum. Comput. Stud..

[28]  N. Srinivasan,et al.  Naturalizing Sense of Agency with a Hierarchical Event-Control Approach , 2014, PloS one.

[29]  James J. Gibson,et al.  The Ecological Approach to Visual Perception: Classic Edition , 2014 .

[30]  David Coyle,et al.  Empirical Evidence for a Diminished Sense of Agency in Speech Interfaces , 2015, CHI.

[31]  Eyal Ofek,et al.  Sparse Haptic Proxy: Touch Feedback in Virtual Environments Using a General Passive Prop , 2017, CHI.

[32]  Bruno Herbelin,et al.  Peripersonal Space: An Index of Multisensory Body–Environment Interactions in Real, Virtual, and Mixed Realities , 2018, Front. ICT.

[33]  Jörg Müller,et al.  Extending the Body for Interaction with Reality , 2017, CHI.

[34]  Mar Gonzalez-Franco,et al.  The uncanny valley of haptics , 2018, Science Robotics.

[35]  Andrew J. Doxon,et al.  Peri-personal space as a prior in coupling visual and proprioceptive signals , 2018, Scientific Reports.

[36]  P. Haggard.  Sense of agency in the human brain , 2017, Nature Reviews Neuroscience.

[37]  Klaus B. Bærentsen,et al.  An activity theory approach to affordance , 2002, NordiCHI '02.

[38]  M. Tsakiris,et al.  Hands only illusion: multisensory integration elicits sense of ownership for body parts but not for non-corporeal objects , 2010, Experimental Brain Research.

[39]  Daniel M. Wolpert,et al.  Forward Models for Physiological Motor Control , 1996, Neural Networks.

[40]  J. Gibson.  The Senses Considered As Perceptual Systems , 1967 .

[41]  Anatole Lécuyer,et al.  Simulating Haptic Feedback Using Vision: A Survey of Research and Applications of Pseudo-Haptic Feedback , 2009, PRESENCE: Teleoperators and Virtual Environments.

[42]  Sriram Subramanian,et al.  Beyond the Libet Clock: Modality Variants for Agency Measurements , 2018, CHI.

[43]  Andrea Serino,et al.  Everyday use of the computer mouse extends peripersonal space representation , 2010, Neuropsychologia.

[44]  M. Slater,et al.  Measuring the Effects through Time of the Influence of Visuomotor and Visuotactile Synchronous Stimulation on a Virtual Body Ownership Illusion , 2014, Perception.

[45]  S. M. Aglioti,et al.  Mere observation of body discontinuity affects perceived ownership and vicarious agency over a virtual hand , 2015, Experimental Brain Research.

[46]  Per Ola Kristensson,et al.  Effects of Hand Representations for Typing in Virtual Reality , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[47]  Takuji Narumi,et al.  Virtual Hand Realism Affects Object Size Perception in Body-Based Scaling , 2019, 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[48]  Karl J. Friston,et al.  Dynamic causal modeling of touch-evoked potentials in the rubber hand illusion , 2016, NeuroImage.

[49]  Clarisse Sieckenius de Souza,et al.  Semiotic engineering: bringing designers and users together at interaction time , 2005, Interact. Comput..

[50]  James D. Hollan,et al.  Direct Manipulation Interfaces , 1985, Hum. Comput. Interact..

[51]  Justin M. Harris,et al.  If I Were You: Perceptual Illusion of Body Swapping , 2008, PloS one.

[52]  M. Slater,et al.  Embodiment in a Child-Like Talking Virtual Body Influences Object Size Perception, Self-Identification, and Subsequent Real Speaking , 2017, Scientific Reports.

[53]  Takuji Narumi,et al.  Controlling fatigue while lifting objects using Pseudo-haptics in a mixed reality space , 2014, 2014 IEEE Haptics Symposium (HAPTICS).

[54]  A. Maravita,et al.  Tools for the body (schema) , 2004, Trends in Cognitive Sciences.

[55]  Jock D. Mackinlay,et al.  The design space of input devices , 1990, CHI '90.

[56]  Heinrich H. Bülthoff,et al.  Welcome to Wonderland: The Influence of the Size and Shape of a Virtual Hand On the Perceived Size and Shape of Virtual Objects , 2013, PloS one.

[57]  Anatole Lécuyer,et al.  The Visual Appearance of User's Avatar Can Influence the Manipulation of Both Real Devices and Virtual Objects , 2007, 2007 IEEE Symposium on 3D User Interfaces.

[58]  Mel Slater,et al.  The Sense of Embodiment in Virtual Reality , 2012, PRESENCE: Teleoperators and Virtual Environments.

[59]  Ronan Boulic,et al.  Characterizing first and third person viewpoints and their alternation for embodied interaction in virtual reality , 2017, PloS one.

[60]  O. Blanke,et al.  Multisensory Mechanisms in Temporo-Parietal Cortex Support Self-Location and First-Person Perspective , 2011, Neuron.

[61]  R. Passingham,et al.  That's My Hand! Activity in Premotor Cortex Reflects Feeling of Ownership of a Limb , 2004, Science.

[62]  Erin Ash,et al.  Priming or Proteus Effect? Examining the Effects of Avatar Race on In-Game Behavior and Post-Play Aggressive Cognition and Affect in Video Games , 2015, Games Cult..

[63]  T. Metzinger,et al.  Full-body illusions and minimal phenomenal selfhood , 2009, Trends in Cognitive Sciences.

[64]  Susanne Bødker,et al.  Through the Interface: A Human Activity Approach To User Interface Design , 1990 .

[65]  A. Michotte.  The perception of causality , 1963 .

[66]  Chi Thanh Vi,et al.  Agency in Mid-air Interfaces , 2017, CHI.

[67]  Jeremy N. Bailenson,et al.  Experiencing Nature: Embodying Animals in Immersive Virtual Environments Increases Inclusion of Nature in Self and Involvement With Nature , 2016, J. Comput. Mediat. Commun..

[68]  Sophie Jörg,et al.  The effect of realism on the virtual hand illusion , 2016, 2016 IEEE Virtual Reality (VR).

[69]  H. Bülthoff,et al.  Merging the senses into a robust percept , 2004, Trends in Cognitive Sciences.

[70]  H. Ehrsson.  The Experimental Induction of Out-of-Body Experiences , 2007, Science.

[71]  Michiteru Kitazaki,et al.  Illusory body ownership of an invisible body interpolated between virtual hands and feet via visual-motor synchronicity , 2018, Scientific Reports.

[72]  Tabitha C. Peck,et al.  Putting yourself in the skin of a black avatar reduces implicit racial bias , 2013, Consciousness and Cognition.

[73]  Valentin Schwind,et al.  Touch with foreign hands: the effect of virtual hand appearance on visual-haptic integration , 2018, SAP.

[74]  Michel Beaudouin-Lafon,et al.  Instrumental interaction: an interaction model for designing post-WIMP user interfaces , 2000, CHI.

[75]  Victor Kaptelinin,et al.  Affordances in HCI: toward a mediated action perspective , 2012, CHI.

[76]  P. C. Malshe.  What is this interaction? , 1994, The Journal of the Association of Physicians of India.

[77]  C. Spence,et al.  Assessing the Role of the ‘Unity Assumption’ on Multisensory Integration: A Review , 2017, Front. Psychol..

[78]  Nicholas P. Holmes,et al.  The body schema and multisensory representation(s) of peripersonal space , 2004, Cognitive Processing.

[79]  Ronald M. Baecker,et al.  Readings in human-computer interaction : a multidisciplinary approach , 1988 .

[80]  Pamela J. Wisniewski,et al.  Over My Hand: Using a Personalized Hand in VR to Improve Object Size Estimation, Body Ownership, and Presence , 2018, SUI.

[81]  Eyal Ofek,et al.  Haptic Retargeting: Dynamic Repurposing of Passive Haptics for Enhanced Virtual Reality Experiences , 2016, CHI.

[82]  Mandayam A. Srinivasan,et al.  Phantom-Based Haptic Interaction with Virtual Objects , 1997, IEEE Computer Graphics and Applications.

[83]  Jonathan D. Cohen,et al.  Rubber hands ‘feel’ touch that eyes see , 1998, Nature.

[84]  Liana E Brown,et al.  Agency and Control for the Integration of a Virtual Tool into the Peripersonal Space , 2011, Perception.

[85]  Carsten Mehring,et al.  Ownership and Agency of an Independent Supernumerary Hand Induced by an Imitation Brain-Computer Interface , 2015, PloS one.

[86]  M Slater,et al.  Offenders become the victim in virtual reality: impact of changing perspective in domestic violence , 2018, Scientific Reports.

[87]  M. Moens,et al.  First Report of the Cereal Cyst Nematode Heterodera latipons on Wheat in Morocco. , 2012, Plant disease.

[88]  M. Arbib,et al.  Tool use and the distalization of the end-effector , 2009, Psychological research.

[89]  Clemens Nylandsted Klokmose,et al.  VIGO: instrumental interaction in multi-surface environments , 2009, CHI.

[90]  F.H. Previc,et al.  Neuropsychological guidelines for aircraft control stations , 2000, IEEE Engineering in Medicine and Biology Magazine.

[91]  D. Norman.  The psychology of everyday things , 1990 .

[92]  M. Slater,et al.  Illusory ownership of a virtual child body causes overestimation of object sizes and implicit attitude changes , 2013, Proceedings of the National Academy of Sciences.

[93]  Jörg Müller,et al.  Control Theoretic Models of Pointing , 2017, ACM Trans. Comput. Hum. Interact..

[94]  Alessandro Farnè,et al.  Auditory peripersonal space in humans. , 2002, Journal of cognitive neuroscience.

[95]  Paul A. Cairns,et al.  First Person vs. Third Person Perspective in Digital Games: Do Player Preferences Affect Immersion? , 2015, CHI.

[96]  P. Haggard,et al.  Voluntary action and conscious awareness , 2002, Nature Neuroscience.

[97]  Mel Slater,et al.  First Person Perspective of Seated Participants Over a Walking Virtual Body Leads to Illusory Agency Over the Walking , 2016, Scientific Reports.

[98]  Maud Marchal,et al.  Elastic-Arm: Human-scale passive haptic feedback for augmenting interaction and perception in virtual environments , 2015, 2015 IEEE Virtual Reality (VR).

[99]  Luca Chittaro,et al.  Mobile Three-Dimensional Maps for Wayfinding in Large and Complex Buildings: Empirical Comparison of First-Person Versus Third-Person Perspective , 2017, IEEE Transactions on Human-Machine Systems.

[100]  C. Spence,et al.  Extending or projecting peripersonal space with tools? Multisensory interactions highlight only the distal and proximal ends of tools , 2004, Neuroscience Letters.

[101]  Brian D. Fisher,et al.  Comparing cursor orientations for mouse, pointer, and pen interaction , 2005, CHI.

[102]  Konrad Paul Kording,et al.  Bayesian integration in sensorimotor learning , 2004, Nature.

[103]  D. Coyle,et al.  The experience of agency in human-computer interactions: a review , 2014, Front. Hum. Neurosci..

[104]  J. J. Gibson.  The theory of affordances , 1977 .

[105]  Inwook Hwang,et al.  AirPiano: Enhancing music playing experience in virtual reality with mid-air haptic feedback , 2017, 2017 IEEE World Haptics Conference (WHC).

[106]  Mar González-Franco,et al.  The contribution of real-time mirror reflections of motor actions on virtual body ownership in an immersive virtual environment , 2010, 2010 IEEE Virtual Reality Conference (VR).

[107]  Alisa Mandrigin,et al.  Bodily ownership and self-location: Components of bodily self-consciousness , 2013, Consciousness and Cognition.

[108]  R. Riener,et al.  Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review , 2012, Psychonomic Bulletin & Review.

[109]  Mel Slater,et al.  Seeing an Embodied Virtual Hand is Analgesic Contingent on Colocation. , 2017, The journal of pain : official journal of the American Pain Society.

[110]  Konrad Paul Kording,et al.  Over my fake body: body ownership illusions for studying the multisensory basis of own-body perception , 2015, Front. Hum. Neurosci..

[111]  B. Hommel,et al.  Body-ownership for actively operated non-corporeal objects , 2015, Consciousness and Cognition.

[112]  Maria V. Sanchez-Vives,et al.  Inducing Illusory Ownership of a Virtual Body , 2009, Front. Neurosci..

[113]  Maria V. Sanchez-Vives,et al.  Changing bodies changes minds: owning another body affects social cognition , 2015, Trends in Cognitive Sciences.

[114]  Mel Slater,et al.  Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[115]  Joan López-Moliner,et al.  The sense of body ownership relaxes temporal constraints for multisensory integration , 2016, Scientific Reports.

[116]  Alessandro Farnè,et al.  Tool-use: An open window into body representation and its plasticity , 2016, Cognitive neuropsychology.

[117]  Mel Slater,et al.  Body ownership causes illusory self-attribution of speaking and influences subsequent real speaking , 2014, Proceedings of the National Academy of Sciences.

[118]  Kasper Hornbæk,et al.  What Is Interaction? , 2017, CHI.

[119]  Ferran Argelaguet,et al.  Touchy : A Visual Approach for Simulating Haptic Effects on Touchscreens , 2019, Front. ICT.

[120]  Jörg Müller,et al.  Ownershift: Facilitating Overhead Interaction in Virtual Reality with an Ownership-Preserving Hand Space Shift , 2018, UIST.

[121]  G. Pellegrino,et al.  Peripersonal space in the brain , 2015, Neuropsychologia.

[122]  D. Burr,et al.  The Ventriloquist Effect Results from Near-Optimal Bimodal Integration , 2004, Current Biology.

[123]  C. Spence,et al.  The Handbook of Multisensory Processing , 2004 .

[124]  Sukhvinder S. Obhi,et al.  Intentional binding and the sense of agency: A review , 2012, Consciousness and Cognition.

[125]  Malin Björnsdotter,et al.  Posterior Cingulate Cortex Integrates the Senses of Self-Location and Body Ownership , 2015, Current Biology.

[126]  Mel Slater,et al.  A Virtual Out-of-Body Experience Reduces Fear of Death , 2017, PloS one.

[127]  Zoubin Ghahramani,et al.  Computational principles of movement neuroscience , 2000, Nature Neuroscience.

[128]  Alice C. Roy,et al.  Tool-use induces morphological updating of the body schema , 2009, Current Biology.

[129]  Yu Sun,et al.  The Effect of Hand Size and Interaction Modality on the Virtual Hand Illusion , 2019, 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[130]  Cristy Ho,et al.  Using Peripersonal Warning Signals to Orient a Driver’s Gaze , 2009, Hum. Factors.

[131]  V. Gallese,et al.  The body beyond the body: expectation of a sensory event is enough to induce ownership over a fake hand , 2013, Proceedings of the Royal Society B: Biological Sciences.

[132]  D. Wolpert,et al.  Abnormalities in the awareness and control of action. , 2000, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[133]  J. Bailenson,et al.  The Proteus Effect: The Effect of Transformed Self-Representation on Behavior , 2007 .

[134]  Mel Slater,et al.  Decreasing Pain Ratings in Chronic Arm Pain Through Changing a Virtual Body: Different Strategies for Different Pain Types. , 2019, The journal of pain : official journal of the American Pain Society.

[135]  Motoyuki Akamatsu,et al.  Multimodal Mouse: A Mouse-Type Device with Tactile and Force Display , 1994, Presence: Teleoperators & Virtual Environments.

[136]  Yutaro Hirao,et al.  Augmented Cross-modality: Translating the Physiological Responses, Knowledge and Impression to Audio-visual Information in Virtual Reality , 2019 .

[137]  Simon Richir,et al.  First- and Third-Person Perspectives in Immersive Virtual Environments: Presence and Performance Analysis of Embodied Users , 2017, Front. Robot. AI.

[138]  Florian Alt,et al.  Looking glass: a field study on noticing interactivity of a shop window , 2012, CHI.

[139]  Francesco Pavani,et al.  Losing One's Hand: Visual-Proprioceptive Conflict Affects Touch Perception , 2009, PloS one.

[140]  Takuji Narumi,et al.  Modifying Perceived Size of a Handled Object through Hand Image Deformation , 2013, PRESENCE: Teleoperators and Virtual Environments.

[141]  Mel Slater,et al.  Virtually Being Einstein Results in an Improvement in Cognitive Task Performance and a Decrease in Age Bias , 2018, Front. Psychol..

[142]  H. Ehrsson,et al.  The moving rubber hand illusion revisited: Comparing movements and visuotactile stimulation to induce illusory ownership , 2014, Consciousness and Cognition.

[143]  S. Schütz-Bosbach,et al.  On agency and body-ownership: Phenomenological and neurocognitive reflections , 2007, Consciousness and Cognition.

[144]  Ye Yuan,et al.  Is the rubber hand illusion induced by immersive virtual reality? , 2010, 2010 IEEE Virtual Reality Conference (VR).

[145]  T. Metzinger,et al.  Video Ergo Sum: Manipulating Bodily Self-Consciousness , 2007, Science.

[146]  Elia Gatti,et al.  Pseudo-Haptic Weight: Changing the Perceived Weight of Virtual Objects By Manipulating Control-Display Ratio , 2019, CHI.

[147]  Donald A. Norman,et al.  THE WAY I SEE ITSignifiers, not affordances , 2008, INTR.

[148]  W. Epstein,et al.  Tool use affects perceived distance, but only when you intend to use it. , 2005, Journal of experimental psychology. Human perception and performance.

[149]  Bernhard Hommel,et al.  Self-perception beyond the body: the role of past agency , 2017, Psychological research.

[150]  H. Henrik Ehrsson,et al.  The Illusion of Owning a Third Arm , 2011, PloS one.

[151]  P. Haggard,et al.  The rubber hand illusion revisited: visuotactile integration and self-attribution. , 2005, Journal of experimental psychology. Human perception and performance.

[152]  M. Tanaka,et al.  Coding of modified body schema during tool use by macaque postcentral neurones. , 1996, Neuroreport.

[153]  Ivan Poupyrev,et al.  The go-go interaction technique: non-linear mapping for direct manipulation in VR , 1996, UIST '96.

[154]  Diego Martinez Plasencia,et al.  A volumetric display for visual, tactile and audio presentation using acoustic trapping , 2019, Nature.

[155]  Paul Kabbash,et al.  The “prince” technique: Fitts' law and selection using area cursors , 1995, CHI '95.

[156]  Karl J. Friston,et al.  Predictions, perception, and a sense of self , 2014, Neurology.

[157]  Maria V. Sanchez-Vives,et al.  First Person Experience of Body Transfer in Virtual Reality , 2010, PloS one.

[158]  C. Gross,et al.  Coding of visual space by premotor neurons. , 1994, Science.

[159]  Daniel Vogel,et al.  The Impact of Control-Display Gain on User Performance in Pointing Tasks , 2008, Hum. Comput. Interact..

[160]  B. Stein,et al.  Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. , 1986, Journal of neurophysiology.

[161]  A. Newen,et al.  Beyond the comparator model: A multifactorial two-step account of agency , 2008, Consciousness and Cognition.

[162]  Donald A. Norman,et al.  Affordance, conventions, and design , 1999, INTR.

[163]  Ana Tajadura-Jiménez,et al.  As Light as your Footsteps: Altering Walking Sounds to Change Perceived Body Weight, Emotional State and Gait , 2015, CHI.

[164]  Justin A. Harris,et al.  The influence of body-ownership cues on tactile sensitivity , 2011, Cognitive neuroscience.

[165]  Miguel A. Nacenta,et al.  Quantitative Measurement of Tool Embodiment for Virtual Reality Input Alternatives , 2019, CHI.

[166]  Ali Israr,et al.  Sensing the future of HCI , 2016, Interactions.

[167]  Ben Shneiderman,et al.  Direct manipulation for comprehensible, predictable and controllable user interfaces , 1997, IUI '97.

[168]  Caitlin Kelleher.  Looking Glass , 2015, SIGCSE.

[169]  Joe Marshall,et al.  Sensory Alignment in Immersive Entertainment , 2019, CHI.