Hand Tracking for Immersive Virtual Reality: Opportunities and Challenges

Hand tracking has become an integral feature of recent generations of immersive virtual reality head-mounted displays. With the widespread adoption of this feature, hardware engineers and software developers are faced with an exciting array of opportunities and a number of challenges, mostly in relation to the human user. In this article, I outline what I see as the main possibilities for hand tracking to add value to immersive virtual reality as well as some of the potential challenges in the context of the psychology and neuroscience of the human user. It is hoped that this paper serves as a roadmap for the development of best practices in the field for the development of subsequent generations of hand tracking and virtual reality technologies.

[1]  Malte F. Jung,et al.  Understanding the Uncanny: Both Atypical Features and Category Ambiguity Provoke Aversion toward Humanlike Robots , 2017, Front. Psychol..

[2]  Eyal Ofek,et al.  Haptic Retargeting: Dynamic Repurposing of Passive Haptics for Enhanced Virtual Reality Experiences , 2016, CHI.

[3]  A. Gordon,et al.  Contribution of tactile information to accuracy in pointing movements , 2001, Experimental Brain Research.

[4]  P. Rochat,et al.  The Uncanny Valley: Existence and Explanations , 2015 .

[5]  Roope Raisamo,et al.  A Survey of Mid-Air Ultrasonic Tactile Feedback , 2019, 2019 IEEE International Symposium on Multimedia (ISM).

[6]  M. Tsakiris,et al.  Beyond the colour of my skin: How skin colour affects the sense of body-ownership , 2012, Consciousness and Cognition.

[7]  Sebastian Möller,et al.  Influence of Hand Tracking as a Way of Interaction in Virtual Reality on User Experience , 2020, 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX).

[8]  J. Kätsyri,et al.  A review of empirical evidence on different uncanny valley hypotheses: support for perceptual mismatch as one road to the valley of eeriness , 2015, Front. Psychol..

[9]  Matthew Heath,et al.  A lower visual field advantage for endpoint stability but no advantage for online movement precision , 2006, Experimental Brain Research.

[10]  T. Howard,et al.  Can Looking at a Hand Make Your Skin Crawl? Peering into the Uncanny Valley for Hands , 2013, Perception.

[11]  Sergei V. Adamovich,et al.  Coordination of reach-to-grasp in physical and haptic-free virtual environments , 2019, Journal of NeuroEngineering and Rehabilitation.

[12]  Yaser Sheikh,et al.  The eyes have it , 2020, ACM Trans. Graph..

[13]  Michael Gaebler,et al.  Controller-Free Hand Tracking for Grab-and-Place Tasks in Immersive Virtual Reality: Design Elements and Their Empirical Study , 2020 .

[14]  H. Brenton,et al.  The Uncanny Valley : does it exist ? , 2005 .

[15]  Mathew Yarossi,et al.  Control of aperture closure during reach-to-grasp movements in immersive haptic-free virtual reality , 2020 .

[16]  M. Goodale,et al.  Superior performance for visually guided pointing in the lower visual field , 2001, Experimental Brain Research.

[17]  Ferran Argelaguet,et al.  The role of interaction in virtual embodiment: Effects of the virtual hand representation , 2016, 2016 IEEE Virtual Reality (VR).

[18]  Gavin Buckingham,et al.  Perceiving and acting upon weight illusions in the absence of somatosensory information. , 2016, Journal of neurophysiology.

[19]  L. Vollero,et al.  Different level of virtualization of sight and touch produces the uncanny valley of avatar’s hand embodiment , 2019, Scientific Reports.

[20]  Mar Gonzalez-Franco,et al.  The uncanny valley of haptics , 2018, Science Robotics.

[21]  Mel Slater,et al.  The Sense of Embodiment in Virtual Reality , 2012, PRESENCE: Teleoperators and Virtual Environments.

[22]  Ellen Poliakoff,et al.  Investigating the uncanny valley for prosthetic hands , 2018, Prosthetics and orthotics international.

[23]  Melvyn A. Goodale,et al.  Real-time vision, tactile cues, and visual form agnosia: removing haptic feedback from a “natural” grasping task induces pantomime-like grasps , 2015, Front. Hum. Neurosci..

[24]  Tessa R. Flack,et al.  Removing Hand Form Information Specifically Impairs Emotion Recognition for Fearful and Angry Body Stimuli , 2019, Perception.

[25]  D. Pélisson,et al.  From Eye to Hand: Planning Goal-directed Movements , 1998, Neuroscience & Biobehavioral Reviews.

[26]  Abeba Birhane,et al.  Algorithmic injustice: a relational ethics approach , 2021, Patterns.

[27]  Mathew Yarossi,et al.  Control of aperture closure during reach-to-grasp movements in immersive haptic-free virtual reality. , 2021, Experimental brain research.

[28]  Susan Goldin-Meadow,et al.  Is Seeing Gesture Necessary to Gesture Like a Native Speaker? , 2016, Psychological science.

[29]  Si Wu,et al.  Asymmetric representations of upper and lower visual fields in egocentric and allocentric references. , 2017, Journal of vision.

[30]  Timnit Gebru,et al.  Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification , 2018, FAT.

[31]  Craig S. Chapman,et al.  What’s limbs got to do with it? Real-world movement correlates with feelings of ownership over virtual arms during object interactions in virtual reality , 2021 .

[32]  Allison M. Okamura,et al.  Fingertip Tactile Devices for Virtual Object Manipulation and Exploration , 2017, CHI.

[33]  Heloir,et al.  The Uncanny Valley , 2019, The Animation Studies Reader.

[34]  Gunter Loffler,et al.  Distinct lower visual field preference for object shape. , 2015, Journal of vision.

[35]  Sofia Seinfeld,et al.  Impact of visuomotor feedback on the embodiment of virtual hands detached from the body , 2020, Scientific reports.

[36]  Kevin C. Dieter,et al.  Kinesthesis Can Make an Invisible Hand Visible , 2014, Psychological science.

[37]  R. Johansson,et al.  Eye–Hand Coordination in Object Manipulation , 2001, The Journal of Neuroscience.

[38]  Daniel Hepperle,et al.  Differences in the Uncanny Valley between Head-Mounted Displays and Monitors , 2020, 2020 International Conference on Cyberworlds (CW).

[39]  Albert H Vette,et al.  Using synchronized eye and motion tracking to determine high-precision eye-movement patterns during object-interaction tasks. , 2018, Journal of vision.

[40]  Joze Guna,et al.  An Analysis of the Precision and Reliability of the Leap Motion Sensor and Its Suitability for Static and Dynamic Tracking , 2014, Sensors.

[41]  M. Pyasik,et al.  Visual appearance of the virtual hand affects embodiment in the virtual hand illusion , 2020, Scientific Reports.

[42]  M. Land.  Vision, eye movements, and natural behavior , 2009, Visual Neuroscience.

[43]  S. Goldin-Meadow,et al.  Why people gesture when they speak , 1998, Nature.

[44]  F. Previc.  Functional specialization in the lower and upper visual fields in humans: Its ecological origins and neurophysiological implications , 1990, Behavioral and Brain Sciences.

[45]  Mark R. Wilson,et al.  A Framework for the Testing and Validation of Simulated Environments in Experimentation and Training , 2020, Frontiers in Psychology.

[46]  Martin Breidt,et al.  Face reality: investigating the Uncanny Valley for virtual faces , 2010, SIGGRAPH ASIA.

[47]  Karl F. MacDorman,et al.  Too real for comfort? Uncanny responses to computer generated faces , 2009, Comput. Hum. Behav..

[48]  Mel Slater,et al.  Enhancing Our Lives with Immersive Virtual Reality , 2016, Front. Robot. AI.

[49]  Gavin P. Lawrence,et al.  Differences in visuomotor control between the upper and lower visual fields , 2005, Experimental Brain Research.

[50]  Jody C. Culham,et al.  fMRI reveals a lower visual field preference for hand actions in human superior parieto-occipital cortex (SPOC) and precuneus , 2013, Cortex.

[51]  P. Kyberd,et al.  Upper- and lower-limb amputees show reduced levels of eeriness for images of prosthetic hands , 2019, Psychonomic bulletin & review.

[52]  A. Takanishi,et al.  Walking in the uncanny valley: importance of the attractiveness on the acceptance of a robot as a working partner , 2015, Front. Psychol..

[53]  Spyros Vosinakis,et al.  Evaluation of visual feedback techniques for virtual grasping with bare hands using Leap Motion and Oculus Rift , 2018, Virtual Reality.

[54]  Michael Wybrow,et al.  Unscripted Retargeting: Reach Prediction for Haptic Retargeting in Virtual Reality , 2021, 2021 IEEE Virtual Reality and 3D User Interfaces (VR).

[55]  P. Boggio,et al.  The influence of skin colour on the experience of ownership in the rubber hand illusion , 2017, Scientific Reports.

[56]  H. Ishiguro,et al.  The thing that should not be: predictive coding and the uncanny valley in perceiving human and humanoid robot actions , 2011, Social cognitive and affective neuroscience.

[57]  T. Ganel,et al.  Grasping Weber’s Law in a Virtual Environment: The Effect of Haptic Feedback , 2020, Frontiers in Psychology.