Depth: the Forgotten Dimension in Multisensory Research

The last quarter of a century has seen a dramatic rise of interest in the spatial constraints on multisensory integration. However, until recently, the majority of this research has investigated integration in the space directly in front of the observer. The space around us, however, extends in three spatial dimensions in the front and to the rear beyond such a limited area. The question to be addressed in this review concerns whether multisensory integration operates according to the same rules throughout the whole of three-dimensional space. The results reviewed here not only show that the space around us seems to be divided into distinct functional regions, but they also suggest that multisensory interactions are modulated by the region of space in which stimuli happen to be presented. We highlight a number of key limitations with previous research in this area, including: (1) The focus on only a very narrow region of two-dimensional space in front of the observer; (2) the use of static stimuli in most research; (3) the study of observers who themselves have been mostly static; and (4) the study of isolated observers. All of these factors may change the way in which the senses interact at any given distance, as can the emotional state/personality of the observer. In summarizing these salient issues, we hope to encourage researchers to consider these factors in their own research in order to gain a better understanding of the spatial constraints on multisensory integration as they affect us in our everyday life.

[1]  Christian Frings,et al.  What a car does to your perception: Distance evaluations differ from within and outside of a car , 2016, Psychonomic bulletin & review.

[2]  Katsumi Minakata,et al.  A tutorial on testing the race model inequality , 2015, Attention, Perception, & Psychophysics.

[3]  O. Blanke,et al.  Peripersonal space as the space of the bodily self , 2015, Cognition.

[4]  C Spence,et al.  On the relative contributions of multisensory integration and crossmodal exogenous spatial attention to multisensory response enhancement. , 2015, Acta psychologica.

[5]  G. D. Iannetti,et al.  Laser-Evoked Vertex Potentials Predict Defensive Motor Actions , 2015, Cerebral cortex.

[6]  Qi Chen,et al.  Enhanced visual dominance in far space , 2015, Experimental Brain Research.

[7]  Olaf Blanke,et al.  The wheelchair as a full-body tool extending the peripersonal space , 2015, Front. Psychol..

[8]  C. Spence,et al.  The close proximity of threat: altered distance perception in the anticipation of pain , 2015, Front. Psychol..

[9]  G. D. Iannetti,et al.  How many peripersonal spaces? , 2015, Neuropsychologia.

[10]  O. Blanke,et al.  Full body action remapping of peripersonal space: The case of walking , 2015, Neuropsychologia.

[11]  T. Pozzo,et al.  Dissociating effect of upper limb non-use and overuse on space and body representations , 2015, Neuropsychologia.

[12]  Justine C. Cléry,et al.  Neuronal bases of peripersonal and extrapersonal spaces, their plasticity and their dynamics: Knowns and unknowns , 2015, Neuropsychologia.

[13]  Ana Tajadura-Jiménez,et al.  Emotion-inducing approaching sounds shape the boundaries of multisensory peripersonal space , 2015, Neuropsychologia.

[14]  H. C. Dijkerman,et al.  Visuo-tactile interactions are dependent on the predictive value of the visual stimulus , 2015, Neuropsychologia.

[15]  C. Spence,et al.  Multisensory interactions in the depth plane in front and rear space: A review , 2015, Neuropsychologia.

[16]  C. Spence Multisensory Flavor Perception , 2015, Cell.

[17]  D. Talsma Predictive coding and multisensory integration: an attentional account of the multisensory mind , 2015, Front. Integr. Neurosci..

[18]  N. Van der Stoep,et al.  Audiovisual integration in near and far space: effects of changes in distance and stimulus effectiveness , 2015, Experimental Brain Research.

[19]  Suliann Ben Hamed,et al.  Impact Prediction by Looming Visual Stimuli Enhances Tactile Detection , 2015, The Journal of Neuroscience.

[20]  Nicholas P. Holmes,et al.  Left or right? Rapid visuomotor coding of hand laterality during motor decisions , 2015, Cortex.

[21]  Elisa Magosso,et al.  Extending peripersonal space representation without tool-use: evidence from a combined behavioral-computational approach , 2015, Front. Behav. Neurosci..

[22]  S. van der Stigchel,et al.  Exogenous spatial attention decreases audiovisual integration , 2014, Attention, Perception, & Psychophysics.

[23]  Isabelle Viaud-Delmon,et al.  Cynophobic Fear Adaptively Extends Peri-Personal Space , 2014, Front. Psychiatry.

[24]  C. Spence,et al.  Effectively responding to tactile stimulation: do homologous cue and effector locations really matter? , 2014, Acta psychologica.

[25]  H. C. Dijkerman,et al.  The influence of object identity on obstacle avoidance reaching behaviour. , 2014, Acta psychologica.

[26]  H. Ehrsson,et al.  Multisensory Representation of the Space Near the Hand , 2014, The Neuroscientist : a review journal bringing neurobiology, neurology and psychiatry.

[27]  C. Spence,et al.  Mapping nociceptive stimuli in a peripersonal frame of reference: Evidence from a temporal order judgment task , 2014, Neuropsychologia.

[28]  Massimiliano Di Luca Light Source Distance Affects Perceived Audiovisual Simultaneity , 2014 .

[29]  S. van der Stigchel,et al.  The influence of vertically and horizontally aligned visual distractors on aurally guided saccadic eye movements , 2014, Experimental Brain Research.

[30]  Donna Lloyd,et al.  In Touch with the Future: The Sense of Touch from Cognitive Neuroscience to Virtual Reality , 2014, PRESENCE: Teleoperators and Virtual Environments.

[31]  Cristy Ho,et al.  Reorienting Driver Attention with Dynamic Tactile Cues , 2014, IEEE Transactions on Haptics.

[32]  S. van der Stigchel,et al.  Exogenous orienting of crossmodal attention in 3-D space: Support for a depth-aware crossmodal attentional system , 2013, Psychonomic Bulletin & Review.

[33]  G. Verni,et al.  Amputation and prosthesis implantation shape body and peripersonal space representations , 2013, Scientific Reports.

[34]  S. van der Stigchel,et al.  Exploring near and far regions of space: Distance-specific visuospatial neglect after stroke , 2013, Journal of clinical and experimental neuropsychology.

[35]  Giovanni Gentile,et al.  A Shared Representation of the Space Near Oneself and Others in the Human Premotor Cortex , 2013, Current Biology.

[36]  Gian Domenico Iannetti,et al.  Better Safe Than Sorry? The Safety Margin Surrounding the Body Is Increased by Anxiety , 2013, The Journal of Neuroscience.

[37]  C. Spence Just how important is spatial coincidence to multisensory integration? Evaluating the spatial rule , 2013, Annals of the New York Academy of Sciences.

[38]  A. Clark Whatever next? Predictive brains, situated agents, and the future of cognitive science. , 2013, The Behavioral and brain sciences.

[39]  M. Bassolino,et al.  Tool-use reshapes the boundaries of body and peripersonal space representations , 2013, Experimental Brain Research.

[40]  C. Spence,et al.  Review: The New Handbook of Multisensory Processing: Sensory Ecology, Behaviour, and Evolution , 2013 .

[41]  G. Pellegrino,et al.  Social Modulation of Peripersonal Space Boundaries , 2013, Current Biology.

[42]  C. Spence,et al.  International Multisensory Research Forum 2012 meeting special issue. , 2013, Multisensory Research.

[43]  Andrea Serino,et al.  Suppression of premotor cortex disrupts motor coding of peripersonal space , 2012, NeuroImage.

[44]  M. Sereno,et al.  Mapping multisensory parietal face and body areas in humans , 2012, Proceedings of the National Academy of Sciences.

[45]  Stella F. Lourenco,et al.  Threat modulates perception of looming visual stimuli , 2012, Current Biology.

[46]  A. Serino,et al.  Dynamic Sounds Capture the Boundaries of Peripersonal Space Representation in Humans , 2012, PloS one.

[47]  Charles Spence,et al.  Drive safely with neuroergonomics , 2012 .

[48]  A. Venneri,et al.  Near and far space neglect: Task sensitivity and anatomical substrates , 2012, Neuropsychologia.

[49]  Paul E. Downing,et al.  Viewpoint (In)dependence of Action Representations: An MVPA Study , 2012, Journal of Cognitive Neuroscience.

[50]  Zachary P. Barnett,et al.  Interactions between the spatial and temporal stimulus factors that influence multisensory integration in human performance , 2012, Experimental Brain Research.

[51]  Nicholas P. Holmes,et al.  Does tool use extend peripersonal space? A review and re-analysis , 2012, Experimental Brain Research.

[52]  C. Spence,et al.  Neuroscience and Biobehavioral Reviews , 2022 .

[53]  M. Murray,et al.  Looming Signals Reveal Synergistic Principles of Multisensory Integration , 2012, The Journal of Neuroscience.

[54]  B. Stein The new handbook of multisensory processes , 2012 .

[55]  C. Spence,et al.  The Colavita Visual Dominance Effect , 2012 .

[56]  N. Holmes,et al.  Keeping the world at hand: rapid visuomotor processing for hand–object interactions , 2012, Experimental Brain Research.

[57]  Zhuanghua Shi,et al.  Non-spatial sounds regulate eye movements and enhance visual search. , 2012, Journal of vision.

[58]  Aaron R. Seitz,et al.  Influences of Multisensory Experience on Subsequent Unisensory Processing , 2011, Front. Psychology.

[59]  Andrea Serino,et al.  Fronto-parietal Areas Necessary for a Multisensory Representation of Peripersonal Space in Humans: An rTMS Study , 2011, Journal of Cognitive Neuroscience.

[60]  C. Spence Tool Use and the Representation of Peripersonal Space in Humans , 2011 .

[61]  Amanda L. Bowen,et al.  Visual signals bias auditory targets in azimuth and depth , 2011, Experimental Brain Research.

[62]  Bettina Forster,et al.  When far is near: ERP correlates of crossmodal spatial interactions between tactile and mirror-reflected visual stimuli , 2011, Neuroscience Letters.

[63]  Valeria I. Petkova,et al.  fMRI Adaptation Reveals a Cortical Mechanism for the Coding of Space Near the Hand , 2011, The Journal of Neuroscience.

[64]  Stella F. Lourenco,et al.  Near space and its relation to claustrophobic fear , 2011, Cognition.

[65]  Charles Spence,et al.  Audiotactile interactions in front and rear space , 2011, Neuroscience & Biobehavioral Reviews.

[66]  Brian T Agganis,et al.  Visual Biasing of Auditory Localization in Azimuth and Depth , 2010, Perceptual and motor skills.

[67]  D. Senkowski,et al.  The multifaceted interplay between attention and multisensory integration , 2010, Trends in Cognitive Sciences.

[68]  Tobias Heed,et al.  Others' Actions Reduce Crossmodal Integration in Peripersonal Space , 2010, Current Biology.

[69]  A. Mouraux,et al.  From the neuromatrix to the pain matrix (and back) , 2010, Experimental Brain Research.

[70]  J. Theeuwes,et al.  Attention and the multiple stages of multisensory integration: A review of audiovisual studies. , 2010, Acta psychologica.

[71]  J. Vroomen,et al.  Perception of intersensory synchrony: A tutorial review , 2010, Attention, perception & psychophysics.

[72]  Brigitte Röder,et al.  Semantic confusion regarding the development of multisensory integration: a practical solution , 2010, The European journal of neuroscience.

[73]  F. Pavani,et al.  Action-specific remapping of peripersonal space , 2010, Neuropsychologia.

[74]  A. Farnè,et al.  Peripersonal Space and Body Schema , 2010 .

[75]  L. Harris,et al.  Space and Time in Perception and Action: Mechanisms of simultaneity constancy , 2010 .

[76]  Hiroaki Ishida,et al.  Shared Mapping of Own and Others' Bodies in Visuotactile Bimodal Area of Monkey Parietal Cortex , 2010, Journal of Cognitive Neuroscience.

[77]  Charles Spence,et al.  Capturing spatial attention with multisensory cues: A review , 2009, Hearing Research.

[78]  Claudio Brozzoli,et al.  Peripersonal space : a multisensory interface for body-objects interactions , 2009 .

[79]  Y. Rossetti,et al.  Coding of Visual Space during Motor Preparation: Approaching Objects Rapidly Modulate Corticospinal Excitability in Hand-Centered Coordinates , 2009, The Journal of Neuroscience.

[80]  A. Serino,et al.  Motor Properties of Peripersonal Space in Humans , 2009, PloS one.

[81]  Cristy Ho,et al.  Using Peripersonal Warning Signals to Orient a Driver’s Gaze , 2009, Hum. Factors.

[82]  Bettina Forster,et al.  An ERP Investigation on Visuotactile Interactions in Peripersonal and Extrapersonal Space: Evidence for the Spatial Rule , 2009, Journal of Cognitive Neuroscience.

[83]  Francesco Pavani,et al.  Grasping actions remap peripersonal space , 2009, Neuroreport.

[84]  Karl J. Friston,et al.  Predictive coding under the free-energy principle , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[85]  Gregor Thut,et al.  Selective integration of auditory-visual looming cues by humans , 2009, Neuropsychologia.

[86]  Andrew J. King,et al.  Visual influences on auditory spatial learning , 2008, Philosophical Transactions of the Royal Society B: Biological Sciences.

[87]  C. Spence,et al.  Multisensory warning signals for event perception and safe driving , 2008 .

[88]  Nicholas P. Holmes,et al.  Multi-Sensory Interactions , 2008 .

[89]  Sidney S. Simon,et al.  Merging of the Senses , 2008, Front. Neurosci..

[90]  Charles Spence,et al.  Capturing spatial attention with multisensory cues , 2008, Psychonomic bulletin & review.

[91]  C. Galletti,et al.  Reaching activity in parietal area V6A of macaque: eye influence on arm activity or retinocentric coding of reaching movements? , 2008, The European journal of neuroscience.

[92]  Cristy Ho,et al.  Multisensory In-Car Warning Signals for Collision Avoidance , 2007, Hum. Factors.

[93]  Paul J. Laurienti,et al.  Modality-specific selective attention attenuates multisensory integration , 2007, Experimental Brain Research.

[94]  Ehud Zohary,et al.  Is That Near My Hand? Multisensory Representation of Peripersonal Space in Human Intraparietal Sulcus , 2007, The Journal of Neuroscience.

[95]  M. Woldorff,et al.  Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration? , 2006, Cerebral cortex.

[96]  Jessie Chen,et al.  Neurophysiology of prehension. I. Posterior parietal cortex and object-oriented hand behaviors. , 2007, Journal of neurophysiology.

[97]  M. Wallace,et al.  Early experience determines how the senses will interact. , 2007, Journal of neurophysiology.

[98]  Dylan F. Cooke,et al.  Parieto-frontal interactions, personal space, and defensive behavior , 2006, Neuropsychologia.

[99]  J. Theeuwes,et al.  Eye movement trajectories and what they tell us , 2006, Neuroscience & Biobehavioral Reviews.

[100]  M. Sereno,et al.  A human parietal face area contains aligned head-centered visual and tactile maps , 2006, Nature Neuroscience.

[101]  Cristy Ho,et al.  Verbal interface design: Do verbal directional cues automatically orient visual spatial attention? , 2006, Comput. Hum. Behav..

[102]  C. Spence,et al.  The differential effect of vibrotactile and auditory cues on visual spatial attention , 2006, Ergonomics.

[103]  Hong Z. Tan,et al.  Using spatial vibrotactile cues to direct visual attention in driving scenes , 2005 .

[104]  C. Spence,et al.  Assessing the effectiveness of various auditory cues in capturing a driver's visual attention. , 2005, Journal of experimental psychology. Applied.

[105]  Marty G. Woldorff,et al.  Selective Attention and Multisensory Integration: Multiple Phases of Effects on the Evoked Brain Activity , 2005, Journal of Cognitive Neuroscience.

[106]  A. Pouget,et al.  Reference frames for representing visual and tactile locations in parietal cortex , 2005, Nature Neuroscience.

[107]  R. Campbell,et al.  Audiovisual Integration of Speech Falters under High Attention Demands , 2005, Current Biology.

[108]  Derek H. Arnold,et al.  Timing sight and sound , 2005, Vision Research.

[109]  Karl J. Friston,et al.  A theory of cortical responses , 2005, Philosophical Transactions of the Royal Society B: Biological Sciences.

[110]  Simon Carlile,et al.  Synchronizing to real events: subjective audiovisual alignment scales with perceived auditory depth and speed of sound. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[111]  Barry E Stein,et al.  Visual Experience Is Necessary for the Development of Multisensory Integration , 2004, The Journal of Neuroscience.

[112]  J. Navarra,et al.  Assessing automaticity in audiovisual speech integration: evidence from the speeded classification task , 2004, Cognition.

[113]  Francesco Pavani,et al.  Multisensory contributions to the 3-D representation of visuotactile peripersonal space in humans: evidence from the crossmodal congruency task , 2004, Journal of Physiology-Paris.

[114]  A. Farnè,et al.  Visuo-tactile representation of near-the-body space , 2004, Journal of Physiology-Paris.

[115]  G. Calvert,et al.  Multisensory integration: methodological approaches and emerging principles in the human brain , 2004, Journal of Physiology-Paris.

[116]  C. Spence,et al.  Crossmodal Space and Crossmodal Attention , 2004 .

[117]  Dylan F. Cooke,et al.  Sensorimotor integration in the precentral gyrus: polysensory neurons and defensive movements. , 2004, Journal of neurophysiology.

[118]  J. Lewald,et al.  Auditory-visual temporal integration as a function of distance: no compensation for sound-transmission time in human perception , 2004, Neuroscience Letters.

[119]  D. Burr,et al.  The Ventriloquist Effect Results from Near-Optimal Bimodal Integration , 2004, Current Biology.

[120]  A. Maravita,et al.  Tools for the body (schema) , 2004, Trends in Cognitive Sciences.

[121]  C. Spence,et al.  Multisensory Integration: Maintaining the Perception of Synchrony , 2003, Current Biology.

[122]  Tirin Moore,et al.  Complex movements evoked by microstimulation of the ventral intraparietal area , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[123]  Yôiti Suzuki,et al.  Implicit estimation of sound-arrival time , 2003, Nature.

[124]  Y. Sugita,et al.  Audiovisual perception: Implicit estimation of sound-arrival time , 2003, Nature.

[125]  H. Kennedy,et al.  Anatomical Evidence of Multimodal Integration in Primate Striate Cortex , 2002, The Journal of Neuroscience.

[126]  A J Van Opstal,et al.  Auditory-visual interactions subserving goal-directed saccades in a complex scene. , 2002, Journal of neurophysiology.

[127]  John J. Foxe,et al.  The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. , 2002, Brain research. Cognitive brain research.

[128]  Daniel V. McGehee,et al.  Collision Warning Timing, Driver Distraction, and Driver Response to Imminent Rear-End Collisions in a High-Fidelity Driving Simulator , 2002, Hum. Factors.

[129]  M. Ernst,et al.  Humans integrate visual and haptic information in a statistically optimal fashion , 2002, Nature.

[130]  E. Ldavas,et al.  Auditory Peripersonal Space in Humans , 2002, Journal of Cognitive Neuroscience.

[131]  Jon Driver,et al.  Reaching with a tool extends visual–tactile interactions into far space: evidence from cross-modal extinction , 2001, Neuropsychologia.

[132]  S. Shimojo,et al.  Sound alters visual evoked potentials in humans , 2001, Neuroreport.

[133]  C. Frith,et al.  Multisensory integration and crossmodal attention effects in the human brain - Response , 2001 .

[134]  L M Ward,et al.  Multisensory integration and crossmodal attention effects in the human brain. , 2001, Science.

[135]  P. Bertelson,et al.  The ventriloquist effect does not depend on the direction of automatic visual attention , 2001, Perception & psychophysics.

[136]  K. Zilles,et al.  Polymodal Motion Processing in Posterior Parietal and Premotor Cortex A Human fMRI Study Strongly Implies Equivalencies between Humans and Monkeys , 2001, Neuron.

[137]  S. Shimojo,et al.  Illusions: What you see is what you hear , 2000, Nature.

[138]  M S Graziano,et al.  Coding the location of the arm by sight. , 2000, Science.

[139]  C. Spence,et al.  Attracting attention to the illusory location of a sound: reflexive crossmodal orienting and ventriloquism , 2000, Neuroreport.

[140]  A Farnè,et al.  Dynamic size‐change of hand peripersonal space following tool use , 2000, Neuroreport.

[141]  P. Bertelson,et al.  The ventriloquist effect does not depend on the direction of deliberate visual attention , 2000, Perception & psychophysics.

[142]  Ronald Melzack,et al.  From the gate to the neuromatrix , 1999, Pain.

[143]  C. Gross,et al.  A neuronal representation of the location of nearby sounds , 1999, Nature.

[144]  T Landis,et al.  Near and far visual space in unilateral neglect , 1998, Annals of neurology.

[145]  F. Previc The neuropsychology of 3-D space. , 1998, Psychological bulletin.

[146]  F. Bremmer,et al.  Spatial invariance of visual receptive fields in parietal cortex neurons , 1997, Nature.

[147]  G Rizzolatti,et al.  The Space Around Us , 1997, Science.

[148]  C. Gross,et al.  Visuospatial properties of ventral premotor cortex. , 1997, Journal of neurophysiology.

[149]  S. Tipper,et al.  Selective Reaching to Grasp: Evidence for Distractor Interference Effects , 1997 .

[150]  G. Rizzolatti,et al.  Coding of peripersonal space in inferior premotor cortex (area F4). , 1996, Journal of neurophysiology.

[151]  C. Gross,et al.  The representation of extrapersonal space: A possible role for bimodal, visual-tactile neurons , 1995 .

[152]  M. Frens,et al.  Spatial and temporal factors determine auditory-visual interactions in human saccadic eye movements , 1995, Perception & psychophysics.

[153]  A. Cowey,et al.  Left visuo-spatial neglect can be worse in far than in near space , 1994, Neuropsychologia.

[154]  T. Lamb,et al.  Amplification and kinetics of the activation steps in phototransduction. , 1993, Biochimica et biophysica acta.

[155]  M. Abbott,et al.  What You See Is What You Are , 1993 .

[156]  A. King,et al.  Multisensory integration. , 1993, Science.

[157]  J. Marshall,et al.  Left neglect for near but not far space in man , 1991, Nature.

[158]  M A Meredith,et al.  Multisensory Integration , 1990 .

[159]  Ernst Pöppel,et al.  Mindworks: Time and Conscious Experience , 1988 .

[160]  D. Baylor,et al.  Spectral sensitivity of human cone photoreceptors , 1987, Nature.

[161]  D. Perani,et al.  Unilateral neglect: Personal and extra-personal , 1986, Neuropsychologia.

[162]  Jeff Miller,et al.  Timecourse of coactivation in bimodal divided attention , 1986, Perception & psychophysics.

[163]  Jeff Miller,et al.  Divided attention: Evidence for coactivation with redundant signals , 1982, Cognitive Psychology.

[164]  G. Rizzolatti,et al.  Afferent properties of periarcuate neurons in macaque monkeys. II. Visual responses , 1981, Behavioural Brain Research.

[165]  Juhani Hyva¨rinen Regional distribution of functions in parietal association area 7 of the monkey , 1981, Brain Research.

[166]  J Hyvärinen,et al.  Regional distribution of functions in parietal association area 7 of the monkey. , 1981, Brain research.

[167]  L Leinonen,et al.  Functional properties of neurones in the posterior part of area 7 in awake monkey. , 1980, Acta physiologica Scandinavica.

[168]  A. J. Hudspeth,et al.  Ionic basis of the receptor potential in a vertebrate hair cell , 1979, Nature.

[169]  Judee K. Burgoon,et al.  TOWARD A THEORY OF PERSONAL SPACE EXPECTATIONS AND THEIR VIOLATIONS , 1976 .

[170]  G. R. Engel,et al.  Visual-Auditory Distance Constancy , 1971, Nature.

[171]  M. Dosey,et al.  Personal space and self-protection. , 1969, Journal of personality and social psychology.

[172]  M. Gardner Proximity image effect in sound localization. , 1968, The Journal of the Acoustical Society of America.

[173]  E. Hall,et al.  The Hidden Dimension , 1970 .

[174]  Robert Sommer,et al.  Invasions of Personal Space , 1966 .

[175]  H. Hediger,et al.  Studies of the psychology and behaviour of captive animals in zoos and circuses. , 1956 .

[176]  J. Gibson,et al.  A theoretical field-analysis of automobile-driving , 1938 .