Flexibility and Stability in Sensory Processing Revealed Using Visual-to-Auditory Sensory Substitution

The classical view of sensory processing involves independent processing in sensory cortices and multisensory integration in associative areas. This hierarchical structure has been challenged by evidence of multisensory responses in sensory areas, and dynamic weighting of sensory inputs in associative areas, thus far reported independently. Here, we used a visual-to-auditory sensory substitution algorithm (SSA) to manipulate the information conveyed by sensory inputs while keeping the stimuli intact. During scan sessions before and after SSA learning, subjects were presented with visual images and auditory soundscapes. The findings reveal 2 dynamic processes. First, crossmodal attenuation of sensory cortices changed direction after SSA learning from visual attenuations of the auditory cortex to auditory attenuations of the visual cortex. Secondly, associative areas changed their sensory response profile from strongest response for visual to that for auditory. The interaction between these phenomena may play an important role in multisensory processing. Consistent features were also found in the sensory dominance in sensory areas and audiovisual convergence in associative area Middle Temporal Gyrus. These 2 factors allow for both stability and a fast, dynamic tuning of the system when required.

[1]  Jung-Kyong Kim,et al.  Generalized Learning of Visual-to-auditory Substitution in Sighted Individuals , 2008 .

[2]  David Raposo,et al.  Dynamic weighting of multisensory stimuli shapes decision-making in rats and humans. , 2013, Journal of vision.

[3]  D. Senkowski,et al.  The multifaceted interplay between attention and multisensory integration , 2010, Trends in Cognitive Sciences.

[4]  G. Vandewalle,et al.  Functional specialization for auditory–spatial processing in the occipital cortex of congenitally blind humans , 2011, Proceedings of the National Academy of Sciences.

[5]  Ankoor S. Shah,et al.  Auditory Cortical Neurons Respond to Somatosensory Stimulation , 2003, The Journal of Neuroscience.

[6]  Amir Amedi,et al.  A Ventral Visual Stream Reading Center Independent of Visual Experience , 2012, Current Biology.

[7]  D. Bavelier,et al.  Cross-modal plasticity: where and how? , 2002, Nature Reviews Neuroscience.

[8]  Joshua E. Motelow,et al.  Negative BOLD with large increases in neuronal activity. , 2008, Cerebral cortex.

[9]  Christopher R Fetsch,et al.  Dynamic Reweighting of Visual and Vestibular Cues during Self-Motion Perception , 2009, The Journal of Neuroscience.

[10]  Joost X. Maier,et al.  Multisensory Integration of Dynamic Faces and Voices in Rhesus Monkey Auditory Cortex , 2005 .

[11]  Lars Muckli,et al.  Cortical Plasticity of Audio–Visual Object Representations , 2008, Cerebral cortex.

[12]  N. Logothetis,et al.  Negative functional MRI response correlates with decreases in neuronal activity in monkey visual area V1 , 2006, Nature Neuroscience.

[13]  Carsten Klingner,et al.  Behavioral correlates of negative BOLD signal changes in the primary somatosensory cortex , 2008, NeuroImage.

[14]  R. Campbell,et al.  Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex , 2000, Current Biology.

[15]  David Poeppel,et al.  Visual speech speeds up the neural processing of auditory speech. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[16]  Uta Noppeney,et al.  Physical and Perceptual Factors Shape the Neural Mechanisms That Integrate Audiovisual Signals in Speech Comprehension , 2011, The Journal of Neuroscience.

[17]  R. Sekuler,et al.  Sound alters visual motion perception , 1997, Nature.

[18]  E. Bullmore,et al.  Response amplification in sensory-specific cortices during crossmodal binding. , 1999, Neuroreport.

[19]  Gideon Rothschild,et al.  Multisensory Integration of Natural Odors and Sounds in the Auditory Cortex , 2011, Neuron.

[20]  Guan-Yu Chen,et al.  Three-Dimensional Reconstruction of Brain-wide Wiring Networks in Drosophila at Single-Cell Resolution , 2011, Current Biology.

[21]  C. Schroeder,et al.  Neuronal Oscillations and Multisensory Interaction in Primary Auditory Cortex , 2007, Neuron.

[22]  L. Benevento,et al.  Auditory-visual interaction in single cells in the cortex of the superior temporal sulcus and the orbital frontal cortex of the macaque monkey , 1977, Experimental Neurology.

[23]  J. Rauschecker,et al.  Preserved Functional Specialization for Spatial Processing in the Middle Occipital Gyrus of the Early Blind , 2010, Neuron.

[24]  M. Beauchamp,et al.  Neural Substrates of Reliability-Weighted Visual-Tactile Multisensory Integration , 2010, Front. Syst. Neurosci..

[25]  M. Hallett,et al.  Functional relevance of cross-modal plasticity in blind humans , 1997, Nature.

[26]  S. Zeki.  Functional specialisation in the visual cortex of the rhesus monkey , 1978, Nature.

[27]  B. Stein,et al.  Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. , 1986, Journal of neurophysiology.

[28]  R. Campbell,et al.  Reading Speech from Still and Moving Faces: The Neural Substrates of Visible Speech , 2003, Journal of Cognitive Neuroscience.

[29]  Amir Amedi,et al.  Reading with Sounds: Sensory Substitution Selectively Activates the Visual Word Form Area in the Blind , 2012, Neuron.

[30]  L. Merabet,et al.  What blindness can tell us about seeing again: merging neuroplasticity and neuroprostheses , 2005, Nature Reviews Neuroscience.

[31]  C. S. Green,et al.  Brain plasticity through the life span: learning to learn and action video games. , 2012, Annual review of neuroscience.

[32]  B. Argall,et al.  Integration of Auditory and Visual Information about Objects in Superior Temporal Sulcus , 2004, Neuron.

[33]  L. Bernstein,et al.  Quantified acoustic–optical speech signal incongruity identifies cortical sites of audiovisual speech processing , 2008, Brain Research.

[34]  T. Powell,et al.  An anatomical study of converging sensory pathways within the cerebral cortex of the monkey. , 1970, Brain : a journal of neurology.

[35]  U. Noppeney,et al.  Perceptual Decisions Formed by Accumulation of Audiovisual Evidence in Prefrontal Cortex , 2010, The Journal of Neuroscience.

[36]  R. Goebel,et al.  Integration of Letters and Speech Sounds in the Human Brain , 2004, Neuron.

[37]  John J. Foxe,et al.  Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping. , 2000, Brain research. Cognitive brain research.

[38]  Emiliano Macaluso,et al.  Spatial attention can modulate audiovisual integration at multiple cortical and subcortical sites , 2009, The European journal of neuroscience.

[39]  Michael S. Beauchamp,et al.  Touch, sound and vision in human superior temporal sulcus , 2008, NeuroImage.

[40]  Amir Amedi,et al.  Negative BOLD in Sensory Cortices During Verbal Memory: A Component in Generating Internal Representations? , 2009, Brain Topography.

[41]  Jeffery A. Jones,et al.  Neural processes underlying perceptual enhancement by visual speech gestures , 2003, Neuroreport.

[42]  Stefanie E. Kuchinsky,et al.  Word intelligibility and age predict visual cortex activity during word listening. , 2012, Cerebral cortex.

[43]  Geraint Rees,et al.  Sound alters activity in human V1 in association with illusory visual perception , 2006, NeuroImage.

[44]  D. J. Felleman,et al.  Distributed hierarchical processing in the primate cerebral cortex. , 1991, Cerebral cortex.

[45]  M. Corbetta,et al.  Control of goal-directed and stimulus-driven attention in the brain , 2002, Nature Reviews Neuroscience.

[46]  E. Adrian.  The Sherrington Lectures. I. Sensory integration. , 1949 .

[47]  N. Logothetis,et al.  Visual modulation of neurons in auditory cortex. , 2008, Cerebral cortex.

[48]  J. Fadiman.  Illusions , 1999 .

[49]  Christopher R Fetsch,et al.  Neural correlates of reliability-based cue weighting during multisensory integration , 2011, Nature Neuroscience.

[50]  Micah M. Murray,et al.  Electrical neuroimaging of memory discrimination based on single-trial multisensory learning , 2012, NeuroImage.

[51]  Christoph Kayser,et al.  Do early sensory cortices integrate cross-modal information? , 2007, Brain Structure and Function.

[52]  Jennifer L. Mozolic,et al.  Cross-modal deactivations during modality-specific selective attention , 2008, BMC neurology.

[53]  L. Shams,et al.  Early Integration and Bayesian Causal Inference in Multisensory Perception , 2012 .

[54]  Shinobu Masaki,et al.  Learning-induced neural plasticity associated with improved identification performance after training of a difficult second-language phonetic contrast , 2003, NeuroImage.

[55]  John J. Foxe,et al.  Multisensory contributions to low-level, ‘unisensory’ processing , 2005, Current Opinion in Neurobiology.

[56]  Tandra Ghose,et al.  Generalization between canonical and non-canonical views in object recognition. , 2013, Journal of vision.

[57]  Masaaki Kawahashi,et al.  Renovation of Journal of Visualization , 2010, J. Vis..

[58]  Amir Amedi,et al.  Extensive Cochleotopic Mapping of Human Auditory Cortical Fields Obtained with Phase-Encoding fMRI , 2011, PloS one.

[59]  Amir Amedi,et al.  Disentangling unisensory and multisensory components in audiovisual integration using a novel multifrequency fMRI spectral analysis , 2010, NeuroImage.

[60]  Jean A. Tkach,et al.  Neural correlates of phonological processing in speech sound disorder: A functional magnetic resonance imaging study , 2011, Brain and Language.

[61]  Karl J. Friston,et al.  Analysis of functional MRI time‐series , 1994, Human Brain Mapping.

[62]  Micah M. Murray,et al.  The Neural Bases of Multisensory Processes , 2011 .

[63]  S. Dehaene,et al.  Cultural Recycling of Cortical Maps , 2007, Neuron.

[64]  T. Flash,et al.  Negative blood oxygenation level dependent homunculus and somatotopic information in primary motor cortex and supplementary motor area , 2012, Proceedings of the National Academy of Sciences.

[65]  S-J Blakemore,et al.  Somatosensory activations during the observation of touch and a case of vision-touch synaesthesia. , 2005, Brain : a journal of neurology.

[66]  L. Merabet,et al.  The plastic human brain cortex. , 2005, Annual review of neuroscience.

[67]  J W Belliveau,et al.  Borders of multiple visual areas in humans revealed by functional magnetic resonance imaging. , 1995, Science.

[68]  U. Noppeney,et al.  Distinct Functional Contributions of Primary Sensory and Association Areas to Audiovisual Integration in Object Categorization , 2010, The Journal of Neuroscience.

[69]  Christoph Kayser,et al.  The Multisensory Nature of Unisensory Cortices: A Puzzle Continued , 2010, Neuron.

[70]  M. Ptito,et al.  Beyond visual, aural and haptic movement perception: hMT+ is activated by electrotactile motion stimulation of the tongue in sighted and in congenitally blind individuals , 2010, Brain Research Bulletin.

[71]  S. Shimojo,et al.  Illusions: What you see is what you hear , 2000, Nature.

[72]  M. Ptito,et al.  Recruitment of the middle temporal area by tactile motion in congenital blindness , 2009, Neuroreport.

[73]  G. Calvert,et al.  Multisensory integration: methodological approaches and emerging principles in the human brain , 2004, Journal of Physiology-Paris.

[74]  P. C. Murphy,et al.  Cerebral Cortex , 2017, Cerebral Cortex.

[75]  G. Glover,et al.  Retinotopic organization in human visual cortex and the spatial precision of functional MRI. , 1997, Cerebral cortex.

[76]  Karl J. Friston,et al.  Statistical parametric maps in functional imaging: A general linear approach , 1994 .

[77]  Micah M. Murray,et al.  The brain uses single-trial multisensory memories to discriminate without awareness , 2005, NeuroImage.

[78]  M. Ernst,et al.  Humans integrate visual and haptic information in a statistically optimal fashion , 2002, Nature.

[79]  M. Sigman,et al.  The neural code for written words: a proposal , 2005, Trends in Cognitive Sciences.

[80]  William M. Stern,et al.  Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex , 2007, Nature Neuroscience.

[81]  Mauro Ursino,et al.  Crossmodal Links between Vision and Touch in Spatial Attention: A Computational Modelling Study , 2009, Comput. Intell. Neurosci..

[82]  Karl J. Friston,et al.  Multisubject fMRI Studies and Conjunction Analyses , 1999, NeuroImage.

[83]  A. Ghazanfar,et al.  Is neocortex essentially multisensory? , 2006, Trends in Cognitive Sciences.

[84]  John J. Foxe,et al.  Auditory-somatosensory multisensory processing in auditory association cortex: an fMRI study. , 2002, Journal of neurophysiology.

[85]  D. Heeger,et al.  Linear Systems Analysis of Functional Magnetic Resonance Imaging in Human V1 , 1996, The Journal of Neuroscience.

[86]  Adam Gazzaley,et al.  Top‐Down Modulation and Normal Aging , 2007, Annals of the New York Academy of Sciences.

[87]  T. James,et al.  An additive-factors design to disambiguate neuronal and areal convergence: measuring multisensory interactions between audio, visual, and haptic sensory streams using fMRI , 2009, Experimental Brain Research.

[88]  Uta Noppeney,et al.  Prior auditory information shapes visual category-selectivity in ventral occipito-temporal cortex , 2010, NeuroImage.

[89]  R. Malach,et al.  Negative BOLD Differentiates Visual Imagery and Perception , 2005, Neuron.

[90]  A. Amedi,et al.  The large-scale organization of "visual" streams emerges without visual experience. , 2012, Cerebral cortex.

[91]  G. Calvert.  Crossmodal processing in the human brain: insights from functional neuroimaging studies. , 2001, Cerebral cortex.

[92]  Á. Pascual-Leone,et al.  The metamodal organization of the brain. , 2001, Progress in brain research.

[93]  D. Poeppel,et al.  Towards a functional neuroanatomy of speech perception , 2000, Trends in Cognitive Sciences.

[94]  Jörn Diedrichsen,et al.  Two Distinct Ipsilateral Cortical Representations for Individuated Finger Movements , 2012, Cerebral cortex.

[95]  Jonathan D. Cohen,et al.  Improved Assessment of Significant Activation in Functional Magnetic Resonance Imaging (fMRI): Use of a Cluster‐Size Threshold , 1995, Magnetic resonance in medicine.

[96]  Matthew W. G. Dye,et al.  Do deaf individuals see better? , 2006, Trends in Cognitive Sciences.

[97]  Y. Yen,et al.  Deactivation of Sensory-Specific Cortex by Cross-Modal Stimuli , 2002, Journal of Cognitive Neuroscience.

[98]  Murray Mm,et al.  Early Integration and Bayesian Causal Inference in Multisensory Perception -- The Neural Bases of Multisensory Processes , 2012 .

[99]  Jung-Kyong Kim,et al.  Tactile–Auditory Shape Learning Engages the Lateral Occipital Complex , 2011, The Journal of Neuroscience.

[100]  E. Bullmore,et al.  Activation of auditory cortex during silent lipreading. , 1997, Science.

[101]  Christoph M. Michel,et al.  Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging , 2004, NeuroImage.

[102]  P. Goldman-Rakic,et al.  Preface: Cerebral Cortex Has Come of Age , 1991 .

[103]  M. Corbetta,et al.  Top-Down Control of Human Visual Cortex by Frontal and Parietal Cortex in Anticipatory Visual Spatial Attention , 2008, The Journal of Neuroscience.

[104]  Peter B. L. Meijer,et al.  An experimental system for auditory image representations , 1992, IEEE Transactions on Biomedical Engineering.

[105]  B. Marks.  THE SHERRINGTON LECTURES , 1970 .