Neural basis of multisensory looming signals

Approaching or looming signals are often related to extremely relevant environmental events (e.g. threats or collisions) making these signals critical for survival. However, the neural network underlying multisensory looming processing is not yet fully understood. Using functional magnetic resonance imaging (fMRI) we identified the neural correlates of audiovisual looming processing in humans: audiovisual looming (vs. receding) signals enhance fMRI-responses in low-level visual and auditory areas plus multisensory cortex (superior temporal sulcus; plus parietal and frontal structures). When characterizing the fMRI-response profiles for multisensory looming stimuli, we found significant enhancements relative to the mean and maximum of unisensory responses in looming-sensitive visual and auditory cortex plus STS. Superadditive enhancements were observed in visual cortex. Subject-specific region-of-interest analyses further revealed superadditive response profiles within all sensory-specific looming-sensitive structures plus bilateral STS for audiovisual looming vs. summed unisensory looming conditions. Finally, we observed enhanced connectivity of bilateral STS with low-level visual areas in the context of looming processing. This enhanced coupling of STS with unisensory regions might potentially serve to enhance the salience of unisensory stimulus features and is accompanied by superadditive fMRI-responses. We suggest that this preference in neural signaling for looming stimuli effectively informs animals to avoid potential threats or collisions.

[1]  P. Mamassian,et al.  Multisensory processing in review: from physiology to behaviour. , 2010, Seeing and perceiving.

[2]  B. Argall,et al.  Unraveling multisensory integration: patchy organization within human STS multisensory cortex , 2004, Nature Neuroscience.

[3]  A. Nobre,et al.  Where and When to Pay Attention: The Neural Systems for Directing Attention to Spatial Locations and to Time Intervals as Revealed by Both PET and fMRI , 1998, The Journal of Neuroscience.

[4]  H. Diener,et al.  On the neural basis of focused and divided attention. , 2005, Brain research. Cognitive brain research.

[5]  W. Ball,et al.  Infant Responses to Impending Collision: Optical and Real , 1971, Science.

[6]  Jochen Kaiser,et al.  Audiovisual Functional Magnetic Resonance Imaging Adaptation Reveals Multisensory Integration Effects in Object-Related Sensory Cortices , 2010, The Journal of Neuroscience.

[7]  Carsten Nicolas Boehler,et al.  On perceived synchrony—neural dynamics of audiovisual illusions and suppressions , 2008, Brain Research.

[8]  John P. Wann,et al.  Reduced Sensitivity to Visual Looming Inflates the Risk Posed by Speeding Vehicles When Children Try to Cross the Road , 2011, Psychological science.

[9]  D. Lewkowicz.  Perception of dynamic and static audiovisual sequences in 3- and 4-month-old infants. , 2008, Child development.

[10]  B. Stein.  Neural mechanisms for synthesizing sensory information and producing adaptive behaviors , 1998, Experimental Brain Research.

[11]  Tömme Noesselt,et al.  Endoscopic eye tracking system for fMRI , 2007, Journal of Neuroscience Methods.

[12]  R. Schlauch,et al.  Duration discrimination and subjective duration for ramped and damped sounds. , 2001, The Journal of the Acoustical Society of America.

[13]  Nikos K. Logothetis,et al.  Auditory looming perception in rhesus monkeys , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[14]  Gregor Thut,et al.  Auditory–Visual Multisensory Interactions in Humans: Timing, Topography, Directionality, and Sources , 2010, The Journal of Neuroscience.

[15]  M. Grassi,et al.  The subjective duration of ramped and damped sounds , 2006, Perception & psychophysics.

[16]  T. Griffiths,et al.  Distinct Mechanisms for Processing Spatial Sequences and Pitch Sequences in the Human Auditory Brain , 2003, The Journal of Neuroscience.

[17]  H. Kennedy,et al.  Anatomical Evidence of Multimodal Integration in Primate Striate Cortex , 2002, The Journal of Neuroscience.

[18]  Gregor Thut,et al.  Selective integration of auditory-visual looming cues by humans , 2009, Neuropsychologia.

[19]  D. Simons,et al.  Moving and looming stimuli capture attention , 2003, Perception & psychophysics.

[20]  Gregory McCarthy,et al.  Polysensory interactions along lateral temporal regions evoked by audiovisual speech. , 2003, Cerebral cortex.

[21]  David Alais,et al.  No direction-specific bimodal facilitation for audiovisual motion detection. , 2004, Brain research. Cognitive brain research.

[22]  J. Rieger,et al.  Audiovisual Temporal Correspondence Modulates Human Multisensory Superior Temporal Sulcus Plus Primary Sensory Cortices , 2007, The Journal of Neuroscience.

[23]  K. Amunts,et al.  Probabilistic maps, morphometry, and variability of cytoarchitectonic areas in the human superior parietal cortex. , 2008, Cerebral cortex.

[24]  P. Morosan,et al.  Human Primary Auditory Cortex: Cytoarchitectonic Subdivisions and Mapping into a Spatial Reference System , 2001, NeuroImage.

[25]  Jon Driver,et al.  Audiovisual synchrony enhances BOLD responses in a brain network including multisensory STS while also enhancing target‐detection performance for both modalities , 2011, Human brain mapping.

[26]  Erich Seifritz,et al.  Looming sounds as warning signals: the function of motion cues. , 2009, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[27]  E. Friedman,et al.  Temporal processing. , 1991, Journal of learning disabilities.

[28]  Asif A. Ghazanfar,et al.  Integration of Bimodal Looming Signals through Neuronal Coherence in the Temporal Lobe , 2008, Current Biology.

[29]  S. Iversen,et al.  Detection of Audio-Visual Integration Sites in Humans by Application of Electrophysiological Criteria to the BOLD Effect , 2001, NeuroImage.

[30]  D. Senkowski,et al.  The multifaceted interplay between attention and multisensory integration , 2010, Trends in Cognitive Sciences.

[31]  T. Stanford,et al.  Multisensory integration: current issues from the perspective of the single neuron , 2008, Nature Reviews Neuroscience.

[32]  U. Noppeney,et al.  Superadditive responses in superior temporal sulcus predict audiovisual benefits in object categorization. , 2010, Cerebral cortex.

[33]  T. Stanford,et al.  Challenges in quantifying multisensory integration: alternative criteria, models, and inverse effectiveness , 2009, Experimental Brain Research.

[34]  Mark W Greenlee,et al.  Neural correlates of coherent audiovisual motion perception. , 2007, Cerebral cortex.

[35]  T. Stanford,et al.  Superadditivity in multisensory integration: putting the computation in context. , 2007, Neuroreport.

[36]  Martin P. Paulus,et al.  Human Neuroscience , 2022 .

[37]  J. Driver,et al.  Looming sounds enhance orientation sensitivity for visual stimuli on the same side as such sounds , 2011, Experimental Brain Research.

[38]  U. Noppeney,et al.  Audiovisual Synchrony Improves Motion Discrimination via Enhanced Connectivity between Early Visual and Auditory Areas , 2010, The Journal of Neuroscience.

[39]  Karl J. Friston,et al.  Psychophysiological and Modulatory Interactions in Neuroimaging , 1997, NeuroImage.

[40]  James A. Caviness,et al.  Persistent Fear Responses in Rhesus Monkeys to the Optical Stimulus of "Looming" , 1962, Science.

[41]  G. DeAngelis,et al.  Multisensory integration: psychophysics, neurophysiology, and computation , 2009, Current Opinion in Neurobiology.

[42]  M. Schönwiesner,et al.  Representation of interaural temporal information from left and right auditory space in the human planum temporale and inferior parietal lobe. , 2005, Cerebral cortex.

[43]  M. Alex Meredith,et al.  Neurons and behavior: the same rules of multisensory integration apply , 1988, Brain Research.

[44]  Yong-Jun Liu,et al.  Neuronal Responses to Looming Objects in the Superior Colliculus of the Cat , 2011, Brain, Behavior and Evolution.

[45]  B. Stein,et al.  The role of anterior ectosylvian cortex in cross-modality orientation and approach behavior , 1996, Experimental Brain Research.

[46]  Murray Mm,et al.  Characterization of Multisensory Integration with fMRI: Experimental Design, Statistical Analysis, and Interpretation -- The Neural Bases of Multisensory Processes , 2012 .

[47]  Paul J. Laurienti,et al.  On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies , 2005, Experimental Brain Research.

[48]  S. Peron,et al.  Spike frequency adaptation mediates looming stimulus selectivity in a collision-detecting neuron , 2009, Nature Neuroscience.

[49]  D. Moore,et al.  Auditory Neuroscience: The Salience of Looming Sounds , 2003, Current Biology.

[50]  Joost X. Maier,et al.  Multisensory Integration of Dynamic Faces and Voices in Rhesus Monkey Auditory Cortex , 2005 .

[51]  J. Driver,et al.  Sound-Induced Enhancement of Low-Intensity Vision: Multisensory Influences on Human Sensory-Specific Cortices and Thalamic Bodies Relate to Perceptual Enhancement of Visual Detection Sensitivity , 2010, The Journal of Neuroscience.

[52]  Jan Gläscher,et al.  Visualization of Group Inference Data in Functional Neuroimaging , 2009, Neuroinformatics.

[53]  T. Takeuchi,et al.  Visual search of expansion and contraction , 1997, Vision Research.

[54]  Audrey R. Nath,et al.  fMRI-Guided Transcranial Magnetic Stimulation Reveals That the Superior Temporal Sulcus Is a Cortical Locus of the McGurk Effect , 2010, The Journal of Neuroscience.

[55]  B. Stein,et al.  The Merging of the Senses , 1993 .

[56]  D. Pandya,et al.  Parietal, temporal, and occipita projections to cortex of the superior temporal sulcus in the rhesus monkey: A retrograde tracer study , 1994, The Journal of comparative neurology.

[57]  Rob Gray,et al.  Looming Auditory Collision Warnings for Driving , 2011, Hum. Factors.

[58]  Chris I. Baker,et al.  Integration of Visual and Auditory Information by Superior Temporal Sulcus Neurons Responsive to the Sight of Actions , 2005, Journal of Cognitive Neuroscience.

[59]  John G. Neuhoff,et al.  Perceptual bias for rising tones , 1998, Nature.

[60]  Gregor Thut,et al.  Preperceptual and Stimulus-Selective Enhancement of Low-Level Human Visual Cortex Excitability by Sounds , 2009, Current Biology.

[61]  A. J. Gabor,et al.  Orientation discrimination sensitivity of single units in cat primary visual cortex , 2004, Experimental Brain Research.

[62]  Brian L Allman,et al.  Do cross-modal projections always result in multisensory integration? , 2008, Cerebral cortex.

[63]  M. Woldorff,et al.  Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration? , 2006, Cerebral cortex.

[64]  L. Benevento,et al.  Auditory-visual interaction in single cells in the cortex of the superior temporal sulcus and the orbital frontal cortex of the macaque monkey , 1977, Experimental Neurology.

[65]  Michael S. Beauchamp,et al.  Statistical criteria in fMRI studies of multisensory integration , 2005, Neuroinformatics.

[66]  C. Frith,et al.  Modulation of human visual cortex by crossmodal spatial attention. , 2000, Science.

[67]  Wei Ji Ma,et al.  Linking neurons to behavior in multisensory perception: A computational review , 2008, Brain Research.

[68]  John G. Neuhoff,et al.  Neural Processing of Auditory Looming in the Human Brain , 2002, Current Biology.

[69]  Karl J. Friston,et al.  Analysis of fMRI Time-Series Revisited , 1995, NeuroImage.

[70]  N. Holmes.  The law of inverse effectiveness in neurons and behaviour: Multisensory integration versus normal variability , 2007, Neuropsychologia.

[71]  C. Carter,et al.  Anterior cingulate cortex and conflict detection: An update of theory and data , 2007, Cognitive, affective & behavioral neuroscience.

[72]  T. Stanford,et al.  Evaluating the Operations Underlying Multisensory Integration in the Cat Superior Colliculus , 2005, The Journal of Neuroscience.

[73]  Simon B. Eickhoff,et al.  A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data , 2005, NeuroImage.

[74]  A. Walker-Andrews,et al.  Auditory-visual perception of changing distance by human infants. , 1985, Child development.

[75]  Uta Noppeney,et al.  The contributions of transient and sustained response codes to audiovisual integration. , 2011, Cerebral cortex.

[76]  B. Seltzer,et al.  Architectonics and cortical connections of the upper bank of the superior temporal sulcus in the rhesus monkey: An analysis in the tangential plane , 2003, The Journal of comparative neurology.

[77]  Joost X. Maier,et al.  Natural, Metaphoric, and Linguistic Auditory Direction Signals Have Distinct Influences on Visual Motion Processing , 2009, The Journal of Neuroscience.

[78]  Valeria I. Petkova,et al.  Integration of visual and tactile signals from the hand in the human brain: an FMRI study. , 2011, Journal of neurophysiology.

[79]  Uta Noppeney,et al.  Characterization of Multisensory Integration with fMRI: Experimental Design, Statistical Analysis, and Interpretation , 2012 .

[80]  D. Hubel,et al.  Receptive fields and functional architecture of monkey striate cortex , 1968, The Journal of physiology.

[81]  Ryan A. Stevenson,et al.  Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects , 2007, Experimental Brain Research.

[82]  David Alais,et al.  A bias for looming stimuli to predominate in binocular rivalry , 2007, Vision Research.

[83]  Massimo Grassi,et al.  Sex Difference in Subjective Duration of Looming and Receding Sounds , 2010, Perception.

[84]  M. Murray,et al.  Looming Signals Reveal Synergistic Principles of Multisensory Integration , 2012, The Journal of Neuroscience.

[85]  R. Poldrack.  Region of interest analysis for fMRI. , 2007, Social cognitive and affective neuroscience.

[86]  Asif A Ghazanfar,et al.  Interactions between the Superior Temporal Sulcus and Auditory Cortex Mediate Dynamic Face/Voice Integration in Rhesus Monkeys , 2008, The Journal of Neuroscience.

[87]  A. Ghazanfar,et al.  Is neocortex essentially multisensory? , 2006, Trends in Cognitive Sciences.

[88]  J. Driver,et al.  Multisensory Interplay Reveals Crossmodal Influences on ‘Sensory-Specific’ Brain Regions, Neural Responses, and Judgments , 2008, Neuron.

[89]  Ryan A. Stevenson,et al.  Audiovisual integration in human superior temporal sulcus: Inverse effectiveness and the neural processing of speech and object recognition , 2009, NeuroImage.

[90]  Marianne Latinus,et al.  Cerebral correlates and statistical criteria of cross-modal face and voice integration. , 2011, Seeing and perceiving.

[91]  S. Hillyard,et al.  Neural Basis of the Ventriloquist Illusion , 2007, Current Biology.

[92]  Asif A Ghazanfar,et al.  Looming Biases in Monkey Auditory Cortex , 2007, The Journal of Neuroscience.

[93]  Asif A Ghazanfar,et al.  Multisensory Integration of Looming Signals by Rhesus Monkeys , 2004, Neuron.

[94]  Michael S Beauchamp,et al.  See me, hear me, touch me: multisensory integration in lateral occipital-temporal cortex , 2005, Current Opinion in Neurobiology.