The Effects of Full-Body Avatar Movement Predictions in Virtual Reality using Neural Networks

Motion tracking technologies and avatars in virtual reality (VR) showing the movements of the own body enable high levels of presence and a strong illusion of body ownership (IBO) – key features of immersive systems and gaming experiences in virtual environments. Previous work suggests using software-based algorithms that can not only compensate system latency but also predict future movements of the user to increase input performance. However, the effects of movement prediction in VR on input performance are largely unknown. In this paper, we investigate neural network-based predictions of full-body avatar movements in two scenarios: In the first study, we used a standardized 2D Fitts’ Law task to examine the information throughput in VR. In the second study, we utilized a full-body VR game to determine the users’ performance. We found that both performance and subjective measures in a standardized 2D Fitts’ law task could not benefit from the predicted avatar movements. In an immersive gaming scenario, however, the perceived accuracy of the own body location improved. Presence and body assessments remained more stable and were higher than during the Fitts’ task. We conclude that machine-learning-based predictions could be used to compensate system-related latency but participants only subjectively benefit under certain conditions.

[1]  Stefan Kopp,et al.  The impact of latency on perceptual judgments and motor performance in closed-loop interaction in virtual reality , 2016, VRST.

[2]  Ye Yuan,et al.  Is the rubber hand illusion induced by immersive virtual reality? , 2010, 2010 IEEE Virtual Reality Conference (VR).

[3]  Joseph J. LaViola,et al.  A discussion of cybersickness in virtual environments , 2000, SGCH.

[4]  Donald C. Wunsch,et al.  Predictive head tracking for virtual reality , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[5]  Mel Slater,et al.  The Sense of Embodiment in Virtual Reality , 2012, PRESENCE: Teleoperators and Virtual Environments.

[6]  Paul Henry Dietz,et al.  The effects of latency and motion blur on touch screen user experience , 2014 .

[7]  Niels Henze,et al.  Up to the Finger Tip: The Effect of Avatars on Mid-Air Pointing Accuracy in Virtual Reality , 2018, CHI PLAY.

[8]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[9]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[10]  Daniel J. Wigdor,et al.  How fast is fast enough?: a study of the effects of latency in direct-touch pointing tasks , 2013, CHI.

[11]  Valentin Schwind,et al.  Touch with foreign hands: the effect of virtual hand appearance on visual-haptic integration , 2018, SAP.

[12]  P. Haggard,et al.  Having a body versus moving your body: How agency structures body-ownership , 2006, Consciousness and Cognition.

[13]  P. Fitts.  The information capacity of the human motor system in controlling the amplitude of movement. , 1954, Journal of experimental psychology.

[14]  K. Hiraki,et al.  Detection of visual feedback delay in active and passive self-body movements , 2010, Experimental Brain Research.

[15]  Michael I. Hill,et al.  Generalizeability of Latency Detection in a Variety of Virtual Environments , 2004 .

[16]  Niels Henze,et al.  PredicTouch: A System to Reduce Touchscreen Latency using Neural Networks and Inertial Measurement Units , 2017, ISS.

[17]  Tabitha C. Peck,et al.  Avatar Embodiment. Towards a Standardized Questionnaire , 2018, Front. Robot. AI.

[18]  Frank Biocca,et al.  The Effect of the Agency and Anthropomorphism on Users' Sense of Telepresence, Copresence, and Social Presence in Virtual Environments , 2003, Presence: Teleoperators & Virtual Environments.

[19]  Cristian Axenie,et al.  Neural Network 3D Body Pose Tracking and Prediction for Motion-to-Photon Latency Compensation in Distributed Virtual Reality , 2019, ICANN.

[20]  Nitish Srivastava,et al.  Improving neural networks by preventing co-adaptation of feature detectors , 2012, ArXiv.

[21]  Markus Funk,et al.  Software-reduced touchscreen latency , 2016, MobileHCI.

[22]  Eve E. Hoggan,et al.  Playing with Tactile Feedback Latency in Touchscreen Interaction: Two Approaches , 2011, INTERACT.

[23]  Tomohisa Asai,et al.  Agency elicits body-ownership: proprioceptive drift toward a synchronously acting external proxy , 2015, Experimental Brain Research.

[24]  Niels Henze,et al.  Improving software-reduced touchscreen latency , 2017, MobileHCI.

[25]  M. Graziano.  Where is my arm? The relative role of vision and proprioception in the neuronal representation of limb position. , 1999, Proceedings of the National Academy of Sciences of the United States of America.

[26]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[27]  Takayuki Ito,et al.  Malleable Embodiment: Changing Sense of Embodiment by Spatial-Temporal Deformation of Virtual Human Body , 2017, CHI.

[28]  C. Moore,et al.  Movement and the Rubber Hand Illusion , 2009, Perception.

[29]  James J. Little,et al.  A Simple Yet Effective Baseline for 3d Human Pose Estimation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[30]  E. Muth,et al.  A questionnaire for the assessment of the multiple dimensions of motion sickness. , 2001, Aviation, space, and environmental medicine.

[31]  Ming Ouhyoung,et al.  On latency compensation and its effects on head-motion trajectories in virtual environments , 2000, The Visual Computer.

[32]  Holger Regenbrecht,et al.  Real and Illusory Interactions Enhance Presence in Virtual Environments , 2002, Presence: Teleoperators & Virtual Environments.

[33]  I. Scott MacKenzie,et al.  Extending Fitts' law to two-dimensional tasks , 1992, CHI.

[34]  P. Haggard,et al.  The rubber hand illusion: Sensitivity and reference frame for body ownership , 2007, Consciousness and Cognition.

[35]  Robert S. Allison,et al.  Tolerance of temporal delay in virtual environments , 2001, Proceedings IEEE Virtual Reality 2001.

[36]  Tomohisa Asai,et al.  Dissociation of agency and body ownership following visuomotor temporal recalibration , 2015, Front. Integr. Neurosci..

[37]  Anoop Gupta,et al.  In the blink of an eye: investigating latency perception during stylus interaction , 2014, CHI.

[38]  Hiroshi Watanabe,et al.  Visual-vestibular conflict induced by virtual reality in humans , 2003, Neuroscience Letters.

[39]  Maria V. Sanchez-Vives,et al.  Inducing Illusory Ownership of a Virtual Body , 2009, Front. Neurosci..

[40]  Pascal Perrier,et al.  Reducing Latency with a Continuous Prediction: Effects on Users' Performance in Direct-Touch Target Acquisitions , 2015, ITS.

[41]  Benoît G. Bardy,et al.  Visually Induced Motion Sickness Predicted by Postural Instability , 2002, Hum. Factors.

[42]  I. Scott MacKenzie,et al.  Lag as a determinant of human performance in interactive systems , 1993, INTERCHI.

[43]  Alisa Mandrigin,et al.  Bodily ownership and self-location: Components of bodily self-consciousness , 2013, Consciousness and Cognition.

[44]  Holger Regenbrecht,et al.  The Experience of Presence: Factor Analytic Insights , 2001, Presence: Teleoperators & Virtual Environments.

[45]  M. Whitton,et al.  Effect of latency on presence in stressful virtual environments , 2003, IEEE Virtual Reality, 2003. Proceedings..

[46]  Ahmet M. Kondoz,et al.  Predicting head trajectories in 360° virtual reality videos , 2017, 2017 International Conference on 3D Immersion (IC3D).

[47]  Niels Henze,et al.  Improving Humans' Ability to Interpret Deictic Gestures in Virtual Reality , 2020, CHI.

[48]  Andriy Pavlovych,et al.  Target following performance in the presence of latency, jitter, and signal dropouts , 2011, Graphics Interface.