Action-perception patterns in virtual ball bouncing: Combating system latency and tracking functional validity

How can we evaluate the spatio-temporal performance of virtual environments (VE) for research use? Here we show that end-to-end latency (ETEL) of VE can strongly damage users' perceptual and perceptuo-motor behaviors and that it can be considered to be the key factor for evaluating face and functional fidelity of a VE. We used a virtual ball-bouncing task as a paradigmatic example. Ball bouncing is known to exhibit attractive and repelling states whose localization in the racket cycle is sufficiently thin to be changed by small variations of ETEL. We first present a simple test-bed to measure the intrinsic ETEL of research-related VE systems. We then report results of a psychophysical ball-bouncing experiment in which ETEL was manipulated. While face validity (i.e., subjective experience) was maintained with relatively high values, the results reveal that the perception-action behavior (performance) was damaged with smaller ETEL values. These results call for action-perception variables in order to test the fidelity of VE systems.

[1]  Mel Slater,et al.  Presence in immersive virtual environments , 1993, Proceedings of IEEE Virtual Reality Annual International Symposium.

[2]  W H Warren,et al.  The Way the Ball Bounces: Visual and Auditory Perception of Elasticity and Control of the Bounce Pass , 1987, Perception.

[3]  Thomas B. Sheridan,et al.  Remote Manipulative Control with Transmission Delay , 1963 .

[4]  Bernard D. Adelstein,et al.  Discrimination of changes in latency during head movement , 1999, HCI.

[5]  J. E. Korteling,et al.  A critical review of validation methods for man-in-the loop simulators , 1999 .

[6]  Michael Erb,et al.  The neural correlates of perceiving one's own movements , 2003, NeuroImage.

[7]  R. Miall,et al.  Visuomotor tracking with delayed visual feedback , 1985, Neuroscience.

[8]  Dagmar Sternad,et al.  The dialogue between data and model: passive stability and relaxation behavior in a ball bouncing task , 2004 .

[9]  Jean-François Lapointe.  Effects of joystick mapping and field-of-view on human performance in virtual walkthroughs , 2002, Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission.

[10]  G. Gescheider.  Psychophysics: The Fundamentals , 1997 .

[11]  Philip N. Day,et al.  Modelling the Effects of Delayed Visual Feedback in Real-Time Operator Control Loops : A Cognitive Perspective , 1999 .

[12]  J. Loomis,et al.  Immersive virtual environment technology as a basic research tool in psychology , 1999, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[13]  Bernard D. Adelstein,et al.  Head Tracking Latency in Virtual Environments: Psychophysics and a Model , 2003 .

[14]  Heinrich H. Bülthoff,et al.  Perception and prediction of simple object interactions , 2007, APGV.

[15]  Thomas B. Sheridan,et al.  Musings on Telepresence and Virtual Presence , 1992, Presence: Teleoperators & Virtual Environments.

[16]  R. Miall,et al.  Adaptation to visual feedback delays in manual tracking: evidence against the Smith Predictor model of human visually guided action , 2006, Experimental Brain Research.

[17]  Wildzunas Rm,et al.  Visual display delay effects on pilot performance. , 1996 .

[18]  Linda Ng Boyle,et al.  Proceedings of the Human Factors and Ergonomics Society , 2006 .

[19]  Bernard D. Adelstein,et al.  Sensor spatial distortion, visual latency, and update rate effects on 3D tracking in virtual environments , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[20]  Sarah S. Chance,et al.  Spatial Updating of Self-Position and Orientation During Real, Imagined, and Virtual Locomotion , 1998 .

[21]  Steve T. Bryson.  Effects of lag and frame rate on various tracking tasks , 1993, Electronic Imaging.

[22]  Gary Bishop,et al.  Latency meter: a device for easily monitoring VE delay , 2002 .

[23]  Jack M. Loomis,et al.  Absolute motion parallax weakly determines visual scale in real and virtual environments , 1995, Electronic Imaging.

[24]  Bernard D. Adelstein,et al.  Dynamic Response of Electromagnetic Spatial Displacement Trackers , 1996, Presence: Teleoperators & Virtual Environments.

[25]  G. M. Gauthier,et al.  Visual vestibular interaction: vestibulo-ocular reflex suppression with head-fixed target fixation , 2004, Experimental Brain Research.

[26]  S. Runeson,et al.  The indispensability of precollision kinematics in the visual perception of relative mass , 1993, Perception & psychophysics.

[27]  Gordon D. Love,et al.  Adaptive modally addressed liquid crystal lenses , 2004, SPIE Optics + Photonics.

[28]  Frederick P. Brooks.  What's Real About Virtual Reality? , 1999, IEEE Computer Graphics and Applications.

[29]  J T Todd,et al.  Visual Perception of Relative Mass in Dynamic Events , 1982, Perception.

[30]  Dave Pape,et al.  Video-Based Measurement of System Latency , 2000 .

[31]  Woodrow Barfield,et al.  Comparison of Human Sensory Capabilities with Technical Specifications of Virtual Environment Equipment , 1995, Presence: Teleoperators & Virtual Environments.

[32]  P Péruch,et al.  Homing in Virtual Environments: Effects of Field of View and Path Layout , 1997, Perception.

[33]  Gavin S. P. Miller,et al.  A real-time low-latency hardware light-field renderer , 1999, SIGGRAPH.

[34]  Brian Mirtich,et al.  Timewarp rigid body simulation , 2000, SIGGRAPH.

[35]  M. Jeannerod,et al.  Defective recognition of one's own actions in patients with schizophrenia. , 2001, The American journal of psychiatry.

[36]  R. Miall,et al.  Digital Object Identifier (DOI) 10.1007/s002219900286 RESEARCH ARTICLE , 2022 .

[37]  Robert S. Allison,et al.  Tolerance of temporal delay in virtual environments , 2001, Proceedings IEEE Virtual Reality 2001.

[38]  M. Tarr,et al.  Virtual reality in behavioral neuroscience and beyond , 2002, Nature Neuroscience.

[39]  Kay M. Stanney,et al.  Virtual environments , 2002 .

[40]  Michael W. Haas,et al.  Postural Instability and Motion Sickness in a Fixed-Base Flight Simulator , 2000, Hum. Factors.

[41]  M. Young,et al.  Discrimination of Changes of Latency during Voluntary Hand Movement of Virtual Objects , 1999 .

[42]  Michael W. Haas,et al.  Virtual and Adaptive Environments: Applications, Implications, and Human Performance Issues , 2003 .

[43]  Scott S. Fisher,et al.  Defining, modeling, and measuring system lag in virtual environments , 1990, Other Conferences.

[44]  S. Flynn.  The Perception of Relative Mass in Physical Collisions , 1994 .

[45]  D. Zeltzer,et al.  Simplifying the Programming of Intelligent Environments , 2009 .

[46]  Michael Venturino,et al.  Performance and head movements using a helmet-mounted display with different sized fields-of-view , 1990 .

[47]  S. Schaal,et al.  One-Handed Juggling: A Dynamical Approach to a Rhythmic Movement Task. , 1996, Journal of motor behavior.

[48]  Wijnand A. IJsselsteijn,et al.  Using Behavioral Realism to Estimate Presence: A Study of the Utility of Postural Responses to Motion Stimuli , 2000, Presence: Teleoperators & Virtual Environments.

[49]  Mel Slater,et al.  Measuring Presence: A Response to the Witmer and Singer Presence Questionnaire , 1999, Presence.

[50]  Patrick Péruch,et al.  Acquisition of Spatial Knowledge Through Visual Exploration of Simulated Environments , 1995 .

[51]  Ronald R. Mourant,et al.  Human Factors Issues in Virtual Environments: A Review of the Literature , 1998, Presence.

[52]  G M Gauthier,et al.  Eye-head movement coordination: vestibulo-ocular reflex suppression with head-fixed target fixation. , 1991, Journal of vestibular research : equilibrium & orientation.

[53]  Thomas A. Stoffregen,et al.  On the nature and evaluation of fidelity in virtual environments: Applications, Implications, and Human Performance , 2019 .

[54]  H. Pick,et al.  A Passive Test of the Held Reafference Hypothesis , 1965, Perceptual and motor skills.

[55]  Annie Luciani,et al.  Visual and Haptic Perception of Object Elasticity in a Virtual Squeezing Event , 2006 .

[56]  Benoît G. Bardy,et al.  Learning new perception–action solutions in virtual ball bouncing , 2007, Experimental Brain Research.

[57]  William Ribarsky,et al.  Evaluation of the effects of frame time variation on VR task performance , 1997, Proceedings of IEEE 1997 Annual International Symposium on Virtual Reality.

[58]  James L. Adams.  An Investigation of the Effects of the Time Lag due to Long Transmission Distances Upon Remote Control. Phase 1; Tracking Experiments , 1961 .

[59]  F. Gaunet,et al.  Virtual environments as a promising tool for investigating human spatial cognition. , 1998 .

[60]  Thanh Giang,et al.  Evaluating the visual fidelity of physically based animations , 2003, ACM Trans. Graph..

[61]  Bernard D. Adelstein,et al.  Discriminability of Prediction Artifacts in a Time-Delayed Virtual Environment , 2000 .

[62]  Michael J. Singer,et al.  Measuring Presence in Virtual Environments: A Presence Questionnaire , 1998, Presence.

[63]  D. Cunningham,et al.  Sensorimotor Adaptation to Violations of Temporal Contiguity , 2001, Psychological science.

[64]  D. Sternad,et al.  Actively tracking ‘passive’ stability in a ball bouncing task , 2003, Brain Research.

[65]  Brian Mirtich,et al.  Impulse-based dynamic simulation of rigid body systems , 1996 .

[66]  Mark R. Mine.  Characterization of End-to-End Delays in Head-Mounted Display Systems , 1993 .

[67]  Thomas A Stoffregen,et al.  Voluntary and involuntary postural responses to imposed optic flow. , 2006, Motor control.

[68]  H. Bülthoff,et al.  Driving in the future: temporal visuomotor adaptation and generalization. , 2001, Journal of vision.

[69]  S. Schaal,et al.  Bouncing a ball: tuning into dynamic stability. , 2001, Journal of experimental psychology. Human perception and performance.

[70]  John Dingliana,et al.  Collisions and perception , 2001, TOGS.