The role of mismatches in the sensory feedback provided to indicate selection within a virtual environment

It is generally understood that virtual reality simulations have a high computational cost. Hence, they rarely can reduce completely all the incoherence within the cross-modal sensory outputs provided. The main research approaches to date have consisted in technically reducing possible mismatches, however minimal research has been conducted so as to analyse their influence on human capabilities. Thus, the objective of this study is to provide further insights to the designers of virtual reality about the negative influence of simulation lags and interesting design implications. To clearly show this, we have investigated the importance of coherent sensory feedback by incorporating time delays and spatial misalignments in the feedback provided by the simulation as a response to participant´s actions to mimic computationally expensive environments. We have also evaluated these misalignments considering two typical interaction setups. In particular, the sensory mismatches influence has been assessed in human factors, such as the sense of presence, task performance and delay perception. Our experimental results indicate that the closer the interaction conditions are to real configurations the higher the sensory requirements are regarding accuracy. The implications of this study offer the designer guidelines to prioritise the reduction of those mismatches in the sensory cues provided depending on the simulations goals.

[1]  Renato S. Pellegrini,et al.  Quality assessment of auditory virtual environments , 2001 .

[2]  David Swapp,et al.  Interaction with co-located haptic feedback in virtual reality , 2006, Virtual Reality.

[3]  Jean Vroomen,et al.  No effect of auditory–visual spatial disparity on temporal recalibration , 2007, Experimental Brain Research.

[4]  Colin Ware,et al.  Eye-hand co-ordination with force feedback , 2000, CHI.

[5]  Karin Coninx,et al.  Are existing metaphors in virtual environments suitable for haptic interaction , 2005 .

[6]  William Ribarsky,et al.  Maintaining usability during 3D placement despite delay , 2003, IEEE Virtual Reality, 2003. Proceedings..

[7]  Caroline Jay,et al.  Quantifying the effects of latency on sensory feedback in distributed virtual environments , 2006 .

[8]  In Lee,et al.  Discrimination of Virtual Environments Under Visual and Haptic Rendering Delays , 2007, 2007 Frontiers in the Convergence of Bioscience and Information Technologies.

[9]  Giuseppe Lipari,et al.  Soft Real-Time Systems: Predictability vs. Efficiency (Series in Computer Science) , 2005 .

[10]  E. Boring.  A History of Experimental Psychology. , 1930 .

[11]  Mary Frecker,et al.  Applying the proximity compatibility and the control‐display compatibility principles to engineering design interfaces , 2006 .

[12]  Charles Spence,et al.  Adaptation to audiotactile asynchrony , 2007, Neuroscience Letters.

[13]  Chang-Sung Jeong,et al.  Collaborative object-oriented visualization environment , 2007, Multimedia Tools and Applications.

[14]  Michael J. Singer,et al.  The Factor Structure of the Presence Questionnaire , 2005, Presence: Teleoperators & Virtual Environments.

[15]  Caroline Jay,et al.  Delayed visual and haptic feedback in a reciprocal tapping task , 2005, First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. World Haptics Conference.

[16]  M Di Luca,et al.  Recalibration of audiovisual simultaneity , 2010 .

[17]  Jeremy N. Bailenson,et al.  Virtual interpersonal touch: Haptic interaction and copresence in collaborative virtual environments , 2008, Multimedia Tools and Applications.

[18]  P. Bertelson,et al.  Recalibration of temporal order perception by exposure to audio-visual asynchrony. , 2004, Brain research. Cognitive brain research.

[19]  I. Hirsh,et al.  Perceived order in different sense modalities. , 1961, Journal of experimental psychology.

[20]  Jack M. Loomis,et al.  Distal Attribution and Presence , 1992, Presence: Teleoperators & Virtual Environments.

[21]  Manuel Ferre.  Haptics: Perception, Devices and Scenarios, 6th International Conference, EuroHaptics 2008, Madrid, Spain, June 10-13, 2008, Proceedings , 2008, EuroHaptics.

[22]  P. Gribble,et al.  Temporal constraints on the McGurk effect , 1996, Perception & psychophysics.

[23]  Raquel Viciana-Abad,et al.  Effects of Co-location and Crossmodal Interaction between Haptic, Auditory and Visual Cues in Presence , 2008, EuroHaptics.

[24]  Ming C. Lin,et al.  Accurate and Fast Proximity Queries Between Polyhedra Using Convex Surface Decomposition , 2001, Comput. Graph. Forum.

[25]  A. Kingstone,et al.  Auditory capture of vision: examining temporal ventriloquism. , 2003, Brain research. Cognitive brain research.

[26]  Mel Slater,et al.  Depth of Presence in Virtual Environments , 1994, Presence: Teleoperators & Virtual Environments.

[27]  Anatole Lécuyer,et al.  The Influence of Spatial Delocation on Perceptual Integration of Vision and Touch , 2006, PRESENCE: Teleoperators and Virtual Environments.

[28]  Robert S. Allison,et al.  Tolerance of temporal delay in virtual environments , 2001, Proceedings IEEE Virtual Reality 2001.

[29]  Linda R. Elliott,et al.  Comparing the effects of visual-auditory and visual-tactile feedback on user performance: a meta-analysis , 2006, ICMI '06.

[30]  Paula J. Durlach,et al.  Effect of Variations in Sensory Feedback on Performance in a Virtual Reaching Task , 2005, Presence: Teleoperators & Virtual Environments.

[31]  C. S. Harris.  Perceptual adaptation to inverted, reversed, and displaced vision. , 1965, Psychological review.

[32]  Antonella De Angeli,et al.  Navigation help in 3D worlds: some empirical evidences on use of sound , 2007, Multimedia Tools and Applications.

[33]  Karon E. MacLean,et al.  The perception of cross-modal simultaneity (or ``the Greenwich Observatory Problem'' revisited) , 2001 .

[34]  I. Scott MacKenzie,et al.  Lag as a determinant of human performance in interactive systems , 1993, INTERCHI.

[35]  Giuseppe Lipari,et al.  Soft Real-Time Systems: Predictability vs. Efficiency , 2010 .

[36]  P. Bertelson,et al.  Adaptation to auditory-visual discordance and ventriloquism in semirealistic situations , 1977 .

[37]  Daniel Västfjäll,et al.  ECOLOGICAL ACOUSTICS AND THE MULTI-MODAL PERCEPTION OF ROOMS: REAL AND UNREAL EXPERIENCES OF AUDITORY-VISUAL VIRTUAL ENVIRONMENTS , 2001 .

[38]  Azzedine Boukerche,et al.  Moderating Simulation Lag in Haptic Virtual Environments , 2006, 39th Annual Simulation Symposium (ANSS'06).

[39]  Yves Rossetti,et al.  Prismatic displacement of vision induces transient changes in the timing of eye-hand coordination , 1993, Perception & psychophysics.

[40]  C. Spence,et al.  Multisensory prior entry. , 2001, Journal of experimental psychology. General.

[41]  Samuel S. Silva,et al.  Head-mounted display versus desktop for 3D navigation in virtual reality: a user study , 2008, Multimedia Tools and Applications.

[42]  Sharif Razzaque,et al.  The hand is slower than the eye: a quantitative exploration of visual dominance over proprioception , 2005, IEEE Proceedings. VR 2005. Virtual Reality, 2005..

[43]  André J. Szameitat,et al.  Behavioral and emotional consequences of brief delays in human-computer interaction , 2009, Int. J. Hum. Comput. Stud..

[44]  Wieslaw R. Woszczyk.  Quality Assessment of Multichannel Sound Recordings , 1993 .

[45]  Gerard Jounghyun Kim,et al.  Increasing the effective egocentric field of view with proprioceptive and tactile feedback , 2004, IEEE Virtual Reality 2004.

[46]  Durand R. Begault,et al.  3-D Sound for Virtual Reality and Multimedia Cambridge , 1994 .

[47]  Yvonne de Kort,et al.  Is This My Hand I See Before Me? The Rubber Hand Illusion in Reality, Virtual Reality, and Mixed Reality , 2006, PRESENCE: Teleoperators and Virtual Environments.

[48]  C. Spence,et al.  Audiovisual synchrony perception for music, speech, and object actions , 2006, Brain Research.

[49]  William Ribarsky,et al.  Managing temporal detail in virtual environments: relating system responsiveness to feedback , 1999, CHI EA '99.

[50]  Mary Frecker,et al.  Impact of response delay and training on user performance with text-based and graphical user interfaces for engineering design , 2007 .

[51]  Günther Theile.  Trends and Activities in the Development of Multichannel Sound Systems , 1993 .

[52]  Anatole Lécuyer,et al.  "Boundary of illusion": an experiment of sensory integration with a pseudo-haptic system , 2001, Proceedings IEEE Virtual Reality 2001.