Multimodal Virtual Environments: Response Times, Attention, and Presence

Multimodal virtual environments (VE) succeed better than single-sensory technologies in creating a sense of presence. We hypothesize that the underlying cognitive mechanism is related to a faster mental processing of multimodal events. Comparing simple detection times of unimodal (auditory, visual, and haptic) events, with bimodal and trimodal combinations, we show that mental processing times are in the following order: unimodal > bimodal > trimodal. Given this processing-speed advantage, multimodal VE users start their cognitive process faster, thus, in a similar exposure time they can pay attention to more informative cues and subtle details in the environment and integrate them creatively. This richer, more complete and coherent experience may contribute to an enhanced sense of presence.

[1]  H. Nagata,et al.  The merging of the senses: integration of subthreshold taste and smell , 2000, Nature Neuroscience.

[2]  S Palmisano,et al.  Global-Perspective Jitter Improves Vection in Central Vision , 2000, Perception.

[3]  J. Pernier,et al.  Dynamics of cortico-subcortical cross-modal operations involved in audio-visual object detection in humans. , 2002, Cerebral cortex.

[4]  Frank Biocca,et al.  Visual Touch in Virtual Environments: An Exploratory Study of Presence, Multimodal Interfaces, and Cross-Modal Sensory Illusions , 2001, Presence: Teleoperators & Virtual Environments.

[5]  Robert Lickliter,et al.  Intersensory redundancy facilitates discrimination of tempo in 3-month-old infants. , 2002, Developmental psychobiology.

[6]  Frederick Bonato,et al.  Visual/vestibular conflict, illusory self-motion, and motion sickness , 2004 .

[7]  Mark H. Draper,et al.  Effects of Image Scale and System Time Delay on Simulator Sickness within Head-Coupled Virtual Environments , 2001, Hum. Factors.

[8]  Thomas B. Sheridan,et al.  Musings on Telepresence and Virtual Presence , 1992, Presence: Teleoperators & Virtual Environments.

[9]  Mel Slater,et al.  A Virtual Presence Counter , 2000, Presence: Teleoperators & Virtual Environments.

[10]  Christopher T. Lovelace,et al.  An irrelevant light enhances auditory detection in humans: a psychophysical analysis of multisensory integration in stimulus detection. , 2003, Brain research. Cognitive brain research.

[11]  Maria V. Sanchez-Vives,et al.  From presence to consciousness through virtual reality , 2005, Nature Reviews Neuroscience.

[12]  C. Spence,et al.  The cost of expecting events in the wrong sensory modality , 2001, Perception & psychophysics.

[13]  Lawrence E Marks,et al.  Brighter noise: Sensory enhancement of perceived loudness by concurrent visual stimulation , 2004, Cognitive, affective & behavioral neuroscience.

[14]  B. Stein,et al.  Enhancement of Perceived Visual Intensity by Auditory Stimuli: A Psychophysical Analysis , 1996, Journal of Cognitive Neuroscience.

[15]  Barry Peterson,et al.  Quantitative Measures of Presence in Virtual Environments: The Roles of Attention and Spatial Comprehension , 1999, Cyberpsychology Behav. Soc. Netw..

[16]  R. Dolan,et al.  The Nose Smells What the Eye Sees Crossmodal Visual Facilitation of Human Olfactory Perception , 2003, Neuron.

[17]  Cristiana Cavina-Pratesi,et al.  Redundant target effect and intersensory facilitation from visual-tactile interactions in simple reaction time , 2002, Experimental Brain Research.

[18]  Miriam Reiner,et al.  The role of haptics in immersive telecommunication environments , 2004, IEEE Transactions on Circuits and Systems for Video Technology.

[19]  C. Frith,et al.  Modulation of human visual cortex by crossmodal spatial attention. , 2000, Science.

[20]  John J. Foxe,et al.  Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. , 2002, Brain research. Cognitive brain research.

[21]  Thomas Rosemeier,et al.  Interaction of vestibular, somatosensory and visual signals for postural control and motion perception under terrestrial and microgravity conditions—a conceptual model , 1998, Brain Research Reviews.

[22]  R J Snowden,et al.  Identification of Visual Stimuli is Improved by Accompanying Auditory Stimuli: The Role of Eye Movements and Sound Location , 2001, Perception.

[23]  Paul Brna,et al.  Presence and Reflection in Training: Support for Learning to Improve Quality Decision-Making Skills under Time Limitations , 2001, Cyberpsychology Behav. Soc. Netw..

[24]  J. Honoré,et al.  Reduction of cutaneous reaction time by directing eyes towards the source of stimulation , 1989, Neuropsychologia.

[25]  Eva Lindh Waterworth,et al.  Focus, Locus, and Sensus: The Three Dimensions of Virtual Experience , 2001, Cyberpsychology Behav. Soc. Netw..

[26]  Michael J. Singer,et al.  Measuring Presence in Virtual Environments: A Presence Questionnaire , 1998, Presence.

[27]  Henry Been-Lirn Duh,et al.  “Conflicting” Motion Cues to the Visual and Vestibular Self- Motion Systems Around 0.06 Hz Evoke Simulator Sickness , 2004, Hum. Factors.

[28]  L. Bernstein,et al.  Audiovisual Speech Binding: Convergence or Association? , 2004 .

[29]  M. Keeling,et al.  Involuntary orienting to sound improves visual perception , 2022 .

[30]  Patrick Haggard,et al.  Vision Modulates Somatosensory Cortical Processing , 2002, Current Biology.

[31]  J. Downar,et al.  A multimodal cortical network for the detection of changes in the sensory environment , 2000, Nature Neuroscience.

[32]  R. Lickliter,et al.  Intersensory Redundancy Guides the Development of Selective Attention, Perception, and Cognition in Infancy , 2004 .

[33]  N. Bolognini,et al.  Enhancement of visual perception by crossmodal visuo-auditory interaction , 2002, Experimental Brain Research.

[34]  Cagatay Basdogan,et al.  An experimental study on the role of touch in shared virtual environments , 2000, TCHI.