Multimodal Integration during Self-Motion in Virtual Reality

This chapter begins by a brief description of some of the different types of simulation tools and techniques that are being used to study self-motion perception, along with some of the advantages and disadvantages of the different interfaces. Subsequently, some of the current empirical work investigating multisensory self-motion perception using these technologies will be summarized, focusing mainly on visual, proprioceptive, and vestibular influences during full-body self-motion through space. Finally, the implications of this research for several applied areas will be briefly described.

[1]  Ranxiao Frances Wang,et al.  Perceiving Real-World Viewpoint Changes , 1998 .

[2]  John M. Hollerbach,et al.  Inertial-Force Feedback for the Treadport Locomotion Interface , 2000, Presence: Teleoperators & Virtual Environments.

[3]  H H Bülthoff,et al.  Integration of depth modules: stereo and shading. , 1988, Journal of the Optical Society of America. A, Optics and image science.

[4]  William H Warren,et al.  Path Integration from Optic Flow and Body Senses in a Homing Task , 2002, Perception.

[5]  Heinrich H. Bülthoff,et al.  Control of a lateral helicopter side-step maneuver on an anthropomorphic robot , 2007 .

[6]  M. Ernst,et al.  Walking Straight into Circles , 2009, Current Biology.

[7]  Frank Vahid,et al.  Enabling nonexpert construction of basic sensor-based systems , 2009, TCHI.

[8]  Michael Kerger,et al.  MPI Motion Simulator: Development and Analysis of a Novel Motion Simulator , 2007 .

[9]  Marc O. Ernst,et al.  Tri-modal integration of visual, tactile and auditory signals for the perception of sequences of events , 2008, Brain Research Bulletin.

[10]  Behrang Keshavarz,et al.  Illusory Self-Motion in Virtual Environments , 2014, Handbook of Virtual Environments, 2nd ed..

[11]  J. Saunders,et al.  Do humans optimally integrate stereo and texture information for judgments of surface slant? , 2003, Vision Research.

[12]  Allen Cheung,et al.  Animal navigation: the difficulty of moving in a straight line , 2007, Biological Cybernetics.

[13]  Heinrich H. Bülthoff,et al.  The contributions of visual flow and locomotor cues to walked distance estimation in a virtual environment , 2007, APGV.

[14]  J. F. Soechting,et al.  Postural readjustments induced by linear motion of visual scenes , 1977, Experimental Brain Research.

[15]  Heinrich H. Bülthoff,et al.  Measurement of instantaneous perceived self-motion using continuous pointing , 2009, Experimental Brain Research.

[16]  Mary C. Whitton,et al.  Evaluation of Reorientation Techniques for Walking in Large Virtual Environments , 2008, 2008 IEEE Virtual Reality Conference.

[17]  David N. Lee,et al.  A Theory of Visual Control of Braking Based on Information about Time-to-Collision , 1976, Perception.

[18]  A. Berthoz,et al.  Contribution of the otoliths to the calculation of linear displacement. , 1989, Journal of neurophysiology.

[19]  M. Tarr,et al.  Virtual reality in behavioral neuroscience and beyond , 2002, Nature Neuroscience.

[20]  Adam R. Richardson,et al.  The effect of feedback training on distance estimation in virtual environments , 2005 .

[21]  John M. Hollerbach,et al.  Design Specifications for the Second Generation Sarcos Treadport Locomotion Interface , 2000, Dynamic Systems and Control: Volume 2.

[22]  G. Paige,et al.  Multiple sensory cues underlying the perception of translation and path. , 2007, Journal of neurophysiology.

[23]  H Mittelstaedt,et al.  The influence of otoliths and somatic graviceptors on angular velocity estimation. , 1996, Journal of vestibular research : equilibrium & orientation.

[24]  B. Mohler,et al.  Calibration of Locomotion due to Visual Motion in a Treadmill-based Virtual Environment , 2005 .

[25]  Konrad Paul Kording,et al.  Bayesian integration in sensorimotor learning , 2004, Nature.

[26]  M. Whitton,et al.  Review of Four Studies on the Use of Physiological Reaction as a Measure of Presence in StressfulVirtual Environments , 2005, Applied psychophysiology and biofeedback.

[27]  Stefan Glasauer,et al.  Idiothetic navigation in Gerbils and Humans , 1991 .

[28]  Betty J. Mohler,et al.  Investigations on the interactions between vision and locomotion using a treadmill virtual environment , 2005, IS&T/SPIE Electronic Imaging.

[29]  Heinrich H. Bülthoff,et al.  Bayesian Models for Seeing Shapes and Depth , 1990 .

[30]  Jennifer L. Campos,et al.  The brain weights body‐based cues higher than vision when estimating walked distances , 2010, The European journal of neuroscience.

[31]  M. Sholl,et al.  The relation between horizontality and rod-and-frame and vestibular navigational performance. , 1989, Journal of experimental psychology. Learning, memory, and cognition.

[32]  Roy A. Ruddle,et al.  The benefits of using a walking interface to navigate virtual environments , 2009, TCHI.

[33]  Luc Van Gool,et al.  Procedural modeling of buildings , 2006, ACM Trans. Graph..

[34]  Alessandro De Luca,et al.  Making virtual walking real: Perceptual evaluation of a new treadmill control algorithm , 2010, TAP.

[35]  Daniel Västfjäll,et al.  Sound Representing Self-Motion in Virtual Environments Enhances Linear Vection , 2008, PRESENCE: Teleoperators and Virtual Environments.

[36]  S. H. Seidman,et al.  Translational motion perception and vestiboocular responses in the absence of non-inertial cues , 2007, Experimental Brain Research.

[37]  Jack M. Loomis,et al.  Measuring Spatial Perception with Spatial Updating and Action , 2008 .

[38]  Jeanine K. Stefanucci,et al.  The Role of Effort in Perceiving Distance , 2003, Psychological science.

[39]  A. Berthoz,et al.  Visuo-vestibular interaction in the reconstruction of travelled trajectories , 2003, Experimental Brain Research.

[40]  Heinrich H. Bülthoff,et al.  Visual Homing Is Possible Without Landmarks: A Path Integration Study in Virtual Reality , 2002, Presence: Teleoperators & Virtual Environments.

[41]  Horst Mittelstaedt,et al.  Idiothetic navigation in humans: estimation of path length , 2001, Experimental Brain Research.

[42]  D. Elliott.  Continuous visual information may be important after all: a failure to replicate Thomson (1983). , 1986, Journal of experimental psychology. Human perception and performance.

[43]  David Waller,et al.  The HIVE: A huge immersive virtual environment for research in spatial cognition , 2007, Behavior research methods.

[44]  Hiroo Iwata,et al.  Walking about virtual environments on an infinite floor , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[45]  Peter J. Werkhoven,et al.  The Effects of Proprioceptive and Visual Feedback on Geographical Orientation in Virtual Environments , 1999, Presence: Teleoperators & Virtual Environments.

[46]  Simon Lessels,et al.  For Efficient Navigational Search, Humans Require Full Physical Movement, but Not a Rich Visual Scene , 2006, Psychological science.

[47]  Jennifer L. Campos,et al.  Multisensory Integration in Speed Estimation During Self-Motion , 2003, Cyberpsychology Behav. Soc. Netw..

[48]  Michael Jenkin,et al.  Humans can use optic flow to estimate distance of travel , 2001, Vision Research.

[49]  W. Berger,et al.  Visual influence on human locomotion , 1997 .

[50]  David N. Lee,et al.  Visual proprioceptive control of standing in human infants , 1974 .

[51]  Richard M Wilkie,et al.  The role of visual and nonvisual information in the control of locomotion. , 2005, Journal of experimental psychology. Human perception and performance.

[52]  J. Loomis,et al.  Body-based senses enhance knowledge of directions in large-scale environments , 2004, Psychonomic bulletin & review.

[53]  William H. Warren,et al.  Optic flow is used to control human walking , 2001, Nature Neuroscience.

[54]  A Berthoz,et al.  The contribution of otoliths and semicircular canals to the perception of two‐dimensional passive whole‐body motion in humans , 1997, The Journal of physiology.

[55]  Julie M. Harris,et al.  Guidance of locomotion on foot uses perceived target location rather than optic flow , 1998, Current Biology.

[56]  Bernhard E. Riecke,et al.  Moving sounds enhance the visually-induced self-motion illusion (circular vection) in virtual reality , 2009, TAP.

[57]  David B. Kaber,et al.  The Utility of a Virtual Reality Locomotion Interface for Studying Gait Behavior , 2007, Hum. Factors.

[58]  J. Rieser,et al.  Visual Perception and the Guidance of Locomotion without Vision to Previously Seen Targets , 1990, Perception.

[59]  Jack M. Loomis,et al.  Limited Field of View of Head-Mounted Displays Is Not the Cause of Distance Underestimation in Virtual Environments , 2004, Presence: Teleoperators & Virtual Environments.

[60]  W. Warren,et al.  The role of central and peripheral vision in perceiving the direction of self-motion , 1992, Perception & psychophysics.

[61]  C. Ellard,et al.  A Comparison of Visual and Nonvisual Sensory Inputs to Walked Distance in a Blind-Walking Task , 2003, Perception.

[62]  Philippe Colantoni,et al.  Virtual Environments with Four or More Spatial Dimensions , 2000, Presence: Teleoperators & Virtual Environments.

[63]  Bob G. Witmer,et al.  Judging Perceived and Traversed Distance in Virtual Environments , 1998, Presence.

[64]  Erik Reinhard,et al.  Do HDR displays support LDR content?: a psychophysical evaluation , 2007, ACM Trans. Graph..

[65]  Markus Lappe,et al.  Discrimination of travel distances from ‘situated’ optic flow , 2003, Vision Research.

[66]  Peter Willemsen,et al.  The Influence of Restricted Viewing Conditions on Egocentric Distance Perception: Implications for Real and Virtual Indoor Environments , 2005, Perception.

[67]  C. Gallistel,et al.  The precision of locomotor odometry in humans , 2009, Experimental Brain Research.

[68]  Mary K. Kaiser,et al.  Perceived Orientation in Physical and Virtual Environments: Changes in Perceived Orientation as a Function of Idiothetic Information Available , 2002, Presence: Teleoperators & Virtual Environments.

[69]  Jennifer L. Campos,et al.  Imagined Self-Motion Differs from Perceived Self-Motion: Evidence from a Novel Continuous Pointing Method , 2009, PloS one.

[70]  Sibylle D. Steck,et al.  Inertial cues do not enhance knowledge of environmental layout , 2003, Psychonomic bulletin & review.

[71]  Sharif Razzaque,et al.  Redirected Walking in Place , 2002, EGVE.

[72]  H. Bülthoff,et al.  Orientation Specificity in Long-Term-Memory for Environmental Spaces , 2007 .

[73]  J. Loomis,et al.  Immersive virtual environment technology as a basic research tool in psychology , 1999, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[74]  H. Bülthoff,et al.  Vision and Action in Virtual Environments: Modern Psychophysics in Spatial Cognition Research , 2001 .

[75]  L. Harris,et al.  Visual and non-visual cues in the perception of linear self motion , 2000, Experimental Brain Research.

[76]  Markus Lappe,et al.  Absolute travel distance from optic flow , 2005, Vision Research.

[77]  J. Loomis,et al.  Visual space perception and visually directed action. , 1992, Journal of experimental psychology. Human perception and performance.

[78]  Anne E. Garing,et al.  Calibration of human locomotion and models of perceptual-motor organization. , 1995, Journal of experimental psychology. Human perception and performance.

[79]  M. Ernst,et al.  Humans integrate visual and haptic information in a statistically optimal fashion , 2002, Nature.

[80]  Jack M. Loomis,et al.  Locomotion Mode Affects the Updating of Objects Encountered During Travel: The Contribution of Vestibular and Proprioceptive Inputs to Path Integration , 1998, Presence.

[81]  Betty J. Mohler,et al.  Calibration of locomotion resulting from visual motion in a treadmill-based virtual environment , 2007, TAP.

[82]  B. Bardy,et al.  Optical modulation of locomotion and energy expenditure at preferred transition speed , 2008, Experimental Brain Research.

[83]  Heinrich H. Bülthoff,et al.  A Bayesian model of the disambiguation of gravitoinertial force by visual cues , 2007, Experimental Brain Research.

[84]  F. Bremmer,et al.  The use of optical velocities for distance discrimination and reproduction during visually simulated self motion , 1999, Experimental Brain Research.

[85]  W. Becker,et al.  Perception of angular displacement without landmarks: evidence for Bayesian fusion of vestibular, optokinetic, podokinesthetic, and cognitive information , 2006, Experimental Brain Research.

[86]  Heinrich H Bülthoff,et al.  Bayesian motion estimation accounts for a surprising bias in 3D vision , 2008, Proceedings of the National Academy of Sciences.

[87]  H. Bülthoff,et al.  Merging the senses into a robust percept , 2004, Trends in Cognitive Sciences.

[88]  Carolina Cruz-Neira,et al.  Surround-Screen Projection-Based Virtual Reality: The Design and Implementation of the CAVE , 2023 .

[89]  G. Allen,et al.  Aging and path integration skill: Kinesthetic and vestibular contributions to wayfinding , 2004, Perception & psychophysics.

[90]  Francine Malouin,et al.  A Treadmill and Motion Coupled Virtual Reality System for Gait Training Post-Stroke , 2006, Cyberpsychology Behav. Soc. Netw..

[91]  D. Sheinberg,et al.  Shape from texture: ideal observers and human psychophysics , 1996 .

[92]  D. H. Warren,et al.  Sensory conflict in judgments of spatial direction , 1969 .

[93]  J. Dichgans,et al.  Visual-Vestibular Interaction: Effects on Self-Motion Perception and Postural Control , 1978 .

[94]  J. Rieser,et al.  Bayesian integration of spatial information. , 2007, Psychological bulletin.

[95]  Alain Berthoz,et al.  Interaction of visual and idiothetic information in a path completion task , 2002, Experimental Brain Research.

[96]  D. Elliott.  Continuous visual information may be important after all: a failure to replicate Thomson (1983) , 1986 .

[97]  Heinrich H. Bülthoff,et al.  Learning System Dynamics: Transfer of Tranining in a Helicopter Hover Simulator , 2008 .

[98]  J. Thomson.  Is continuous visual monitoring necessary in visually guided locomotion? , 1983, Journal of experimental psychology. Human perception and performance.

[99]  Jack M. Loomis,et al.  Visual perception of egocentric distance in real and virtual environments. , 2003 .

[100]  John M. Hollerbach,et al.  Slope Display on a Locomotion Interface , 1999, ISER.

[101]  Thomas Banton,et al.  The Perception of Walking Speed in a Virtual Environment , 2005, Presence: Teleoperators & Virtual Environments.

[102]  Peter Willemsen,et al.  Does the Quality of the Computer Graphics Matter when Judging Distances in Visually Immersive Environments? , 2004, Presence: Teleoperators & Virtual Environments.

[103]  Laura F. Fox,et al.  Self-motion perception during locomotor recalibration: more than meets the eye. , 2005, Journal of experimental psychology. Human perception and performance.

[104]  S S Fukusima,et al.  Visual perception of egocentric distance as assessed by triangulation. , 1997, Journal of experimental psychology. Human perception and performance.

[105]  David Waller,et al.  Correcting distance estimates by interacting with immersive virtual environments: effects of task and available sensory information. , 2008, Journal of experimental psychology. Applied.

[106]  J M Flach,et al.  Sources of optical information useful for perception of speed of rectilinear self-motion. , 1990, Journal of experimental psychology. Human perception and performance.

[107]  Sarah S. Chance,et al.  Spatial Updating of Self-Position and Orientation During Real, Imagined, and Virtual Locomotion , 1998 .

[108]  R. Wehner,et al.  The Ant Odometer: Stepping on Stilts and Stumps , 2006, Science.

[109]  Heinrich H. Bülthoff,et al.  Simulating believable forward accelerations on a stewart motion platform , 2010, TAP.

[110]  Alan L. YuilleDivision.  A Bayesian Framework for the Integration of Visual Modules , 1996 .

[111]  Daniel J. Hannon,et al.  Direction of self-motion is perceived from optical flow , 1988, Nature.

[112]  G. DeAngelis,et al.  Neural correlates of multisensory cue integration in macaque MSTd , 2008, Nature Neuroscience.

[113]  D. H. Warren,et al.  Immediate perceptual response to intersensory discrepancy. , 1980, Psychological bulletin.

[114]  Heinrich H. Bülthoff,et al.  A psychophysically calibrated controller for navigating through large environments in a limited free-walking space , 2008, VRST '08.

[115]  Heinrich H. Bülthoff,et al.  Visualization and (Mis)Perceptions in Virtual Reality , 2007 .

[116]  Sholl Mj,et al.  The relation between horizontality and rod-and-frame and vestibular navigational performance. , 1989 .

[117]  Sharif Razzaque,et al.  Redirected Walking , 2001, Eurographics.

[118]  Rudy Darken,et al.  The omni-directional treadmill: a locomotion device for virtual worlds , 1997, UIST '97.

[119]  Jennifer L. Campos,et al.  Bayesian integration of visual and vestibular signals for heading. , 2009, Journal of vision.

[120]  Heinrich H. Bülthoff,et al.  Gait Parameters while Walking in a Head-mounted Display Virtual Environment and the Real World , 2007, EGVE.

[121]  A Berthoz,et al.  Spatial memory of body linear displacement: what is being stored? , 1995, Science.

[122]  Heinrich H. Bülthoff,et al.  Working Memory in Wayfinding - A Dual Task Experiment in a Virtual City , 2008, Cogn. Sci..

[123]  Hiroo Iwata,et al.  Path Reproduction Tests Using a Torus Treadmill , 1999, Presence.

[124]  Jean Laurens,et al.  Bayesian processing of vestibular information , 2007, Biological Cybernetics.

[125]  Heinrich H. Bülthoff,et al.  A High-End Virtual Reality Setup for the Study of Mental Rotations , 2008, PRESENCE: Teleoperators and Virtual Environments.

[126]  Jennifer L. Campos,et al.  Multisensory integration in the estimation of relative path length , 2003, Experimental Brain Research.

[127]  William H. Warren,et al.  Wormholes in Virtual Reality: What spatial knowledge is learned for navigation? , 2010 .

[128]  Christopher R Fetsch,et al.  Dynamic Reweighting of Visual and Vestibular Cues during Self-Motion Perception , 2009, The Journal of Neuroscience.

[129]  Jennifer L. Campos,et al.  The Contributions of Static Visual Cues, Nonvisual Cues, and Optic Flow in Distance Estimation , 2004, Perception.

[130]  Betty J. Mohler,et al.  Visual flow influences gait transition speed and preferred walking speed , 2007, Experimental Brain Research.

[131]  H. Bülthoff,et al.  Spatial updating in virtual reality: the sufficiency of visual information , 2007, Psychological research.

[132]  Heinrich H. Bülthoff,et al.  The role of attention on the integration of visual and inertial cues , 2009, Experimental Brain Research.

[133]  W. Becker,et al.  Estimation of self-turning in the dark: comparison between active and passive rotation , 1999, Experimental Brain Research.

[134]  John J. Rieser,et al.  The recalibration of rotational locomotion , 1999 .

[135]  David Waller,et al.  The role of body-based sensory information in the acquisition of enduring spatial representations , 2007, Psychological research.

[136]  A Berthoz,et al.  Spatial memory and path integration studied by self-driven passive linear displacement. I. Basic properties. , 1997, Journal of neurophysiology.

[137]  W. Berger,et al.  Visual influence on human locomotion Modulation to changes in optic flow , 1997, Experimental Brain Research.

[138]  D. Burr,et al.  The Ventriloquist Effect Results from Near-Optimal Bimodal Integration , 2004, Current Biology.