Integration of visual information in auditory cortex promotes auditory scene analysis through multisensory binding

How and where in the brain audio-visual signals are bound to create multimodal objects remains unknown. One hypothesis is that temporal coherence between dynamic multisensory signals provides a mechanism for binding stimulus features across sensory modalities in early sensory cortex. Here we report that temporal coherence between auditory and visual streams enhances spiking representations in auditory cortex. We demonstrate that when a visual stimulus is temporally coherent with one sound in a mixture, the neural representation of that sound is enhanced. Supporting the hypothesis that these changes represent a neural correlate of multisensory binding, the enhanced neural representation extends to stimulus features other than those that bind auditory and visual streams. These data provide evidence that early cross-sensory binding provides a bottom-up mechanism for the formation of cross-sensory objects and that one role for multisensory binding in auditory cortex is to support auditory scene analysis.

[1]  Adrian K. C. Lee,et al.  Auditory selective attention is enhanced by a task-irrelevant temporally coherent visual stimulus in human listeners , 2015, eLife.

[2]  S. Hughes,et al.  Temporal Framing of Thalamic Relay-Mode Firing by Phasic Inhibition during the Alpha Rhythm , 2009, Neuron.

[3]  Guglielmo Foffani,et al.  PSTH-based classification of sensory stimuli using ensembles of single neurons , 2004, Journal of Neuroscience Methods.

[4]  Thilo Womelsdorf,et al.  A Role of Phase-Resetting in Coordinating Large Scale Neural Networks During Attention and Goal-Directed Behavior , 2016, Front. Syst. Neurosci..

[5]  Gregory Hickok,et al.  An fMRI Study of Audiovisual Speech Perception Reveals Multisensory Interactions in Auditory Cortex , 2013, PloS one.

[6]  Michael J. Crosse,et al.  Eye Can Hear Clearly Now: Inverse Effectiveness in Natural Audiovisual Speech Processing Relies on Long-Term Crossmodal Temporal Integration , 2016, The Journal of Neuroscience.

[7]  D. Poeppel,et al.  Phase Patterns of Neuronal Responses Reliably Discriminate Speech in Human Auditory Cortex , 2007, Neuron.

[8]  Pablo Fuentealba,et al.  Cell Type-Specific Tuning of Hippocampal Interneuron Firing during Gamma Oscillations In Vivo , 2007, The Journal of Neuroscience.

[9]  W. H. Sumby,et al.  Visual contribution to speech intelligibility in noise , 1954 .

[10]  Huriye Atilgan,et al.  The role of spectral cues in timbre discrimination by ferrets and humans. , 2015, The Journal of the Acoustical Society of America.

[11]  Huriye Atilgan,et al.  Acute Inactivation of Primary Auditory Cortex Causes a Sound Localisation Deficit in Ferrets , 2017, PloS one.

[12]  H. Scheich,et al.  Multisensory processing via early cortical stages: Connections of the primary auditory cortical field with other sensory systems , 2006, Neuroscience.

[13]  Italo Masiello,et al.  Architecture and callosal connections of visual areas 17, 18, 19 and 21 in the ferret (Mustela putorius). , 2002, Cerebral cortex.

[14]  S. Shamma,et al.  Temporal Coherence in the Perceptual Organization and Cortical Representation of Auditory Scenes , 2009, Neuron.

[15]  Kai Lu,et al.  Temporal coherence structure rapidly shapes neuronal interactions , 2017, Nature Communications.

[16]  James A. O'Sullivan,et al.  Evidence for Neural Computations of Temporal Coherence in an Auditory Scene and Their Enhancement during Active Listening , 2015, The Journal of Neuroscience.

[17]  Michael J. Crosse,et al.  Congruent Visual Speech Enhances Cortical Entrainment to Continuous Auditory Speech in Noise-Free Conditions , 2015, The Journal of Neuroscience.

[18]  Rachel N. Denison,et al.  Temporal Structure and Complexity Affect Audio-Visual Correspondence Detection , 2013, Front. Psychology.

[19]  R. Metherate,et al.  Spectral integration in primary auditory cortex: Laminar processing of afferent input, in vivo and in vitro , 2005, Neuroscience.

[20]  T. Moore,et al.  Neural Mechanisms of Selective Visual Attention. , 2017, Annual review of psychology.

[21]  C. Gray,et al.  Cellular Mechanisms Contributing to Response Variability of Cortical Neurons In Vivo , 1999, The Journal of Neuroscience.

[22]  D. Poeppel,et al.  Auditory Cortex Tracks Both Auditory and Visual Stimulus Dynamics Using Low-Frequency Neuronal Phase Modulation , 2010, PLoS biology.

[23]  Stefano Panzeri,et al.  Visual Enhancement of the Information Representation in Auditory Cortex , 2010, Current Biology.

[24]  N. Logothetis,et al.  Visual modulation of neurons in auditory cortex. , 2008, Cerebral cortex.

[25]  Christoph Kayser,et al.  Natural asynchronies in audiovisual communication signals regulate neuronal multisensory interactions in voice-sensitive cortex , 2014, Proceedings of the National Academy of Sciences.

[26]  J. B. Levitt,et al.  Retinotopic organization of ferret suprasylvian cortex , 2006, Visual Neuroscience.

[27]  Henning Scheich,et al.  Neuronal activity in primate auditory cortex during the performance of audiovisual tasks , 2015, The European journal of neuroscience.

[28]  Jennifer K. Bizley,et al.  Multiplexed and multivariate representations of sound identity during perceptual constancy , 2017, bioRxiv.

[29]  Michael Okun,et al.  The Subthreshold Relation between Cortical Local Field Potential and Neuronal Firing Unveiled by Intracellular Recordings in Awake Rats , 2010, The Journal of Neuroscience.

[30]  Kerry M. M. Walker,et al.  Interdependent Encoding of Pitch, Timbre, and Spatial Location in Auditory Cortex , 2009, The Journal of Neuroscience.

[31]  Gregor Thut,et al.  Lip movements entrain the observers’ low-frequency brain oscillations to facilitate speech intelligibility , 2016, eLife.

[32]  Xiao-Jing Wang Neurophysiological and computational principles of cortical rhythms in cognition. , 2010, Physiological reviews.

[33]  Arne D. Ekstrom,et al.  Brain Oscillations Control Timing of Single-Neuron Activity in Humans , 2007, The Journal of Neuroscience.

[34]  Asif A. Ghazanfar,et al.  The Natural Statistics of Audiovisual Speech , 2009, PLoS Comput. Biol..

[35]  Asif A Ghazanfar,et al.  Dynamic faces speed up the onset of auditory cortical spiking responses during vocal detection , 2013, Proceedings of the National Academy of Sciences.

[36]  C. Schroeder,et al.  The Spectrotemporal Filter Mechanism of Auditory Selective Attention , 2013, Neuron.

[37]  J. Obleser,et al.  Frequency modulation entrains slow neural oscillations and optimizes human listening behavior , 2012, Proceedings of the National Academy of Sciences.

[38]  R. Freeman,et al.  Neurometabolic coupling in cerebral cortex reflects synaptic more than spiking activity , 2007, Nature Neuroscience.

[39]  I. Nelken,et al.  Physiological and Anatomical Evidence for Multisensory Interactions in Auditory Cortex , 2006, Cerebral cortex.

[40]  P. Schyns,et al.  Cracking the Code of Oscillatory Activity , 2011, PLoS biology.

[41]  W. Klimesch,et al.  EEG alpha oscillations: The inhibition–timing hypothesis , 2007, Brain Research Reviews.

[42]  Jennifer K Bizley,et al.  Where are multisensory signals combined for perceptual decision-making? , 2016, Current Opinion in Neurobiology.

[43]  Joost X. Maier,et al.  Multisensory Integration of Dynamic Faces and Voices in Rhesus Monkey Auditory Cortex , 2005 .

[44]  Gregory B. Cogan,et al.  Visual Input Enhances Selective Speech Envelope Tracking in Auditory Cortex at a “Cocktail Party” , 2013, The Journal of Neuroscience.

[45]  J. Peelle,et al.  Prediction and constraint in audiovisual speech perception , 2015, Cortex.

[46]  R. Desimone,et al.  Neural mechanisms of selective visual attention. , 1995, Annual review of neuroscience.

[47]  Adrian K. C. Lee,et al.  Defining Auditory-Visual Objects: Behavioral Tests and Physiological Mechanisms , 2016, Trends in Neurosciences.

[48]  Charles H. Brown,et al.  The Influence of Natural Scene Dynamics on Auditory Cortical Activity , 2010, The Journal of Neuroscience.

[49]  B. Postle,et al.  Top-down control of the phase of alpha-band oscillations as a mechanism for temporal prediction , 2015, Proceedings of the National Academy of Sciences.

[50]  Kerry M. M. Walker,et al.  Spectral timbre perception in ferrets: discrimination of artificial vowels under different listening conditions. , 2013, The Journal of the Acoustical Society of America.

[51]  C. Schroeder,et al.  Neuronal Oscillations and Multisensory Interaction in Primary Auditory Cortex , 2007, Neuron.

[52]  A. Puce,et al.  Neuronal oscillations and visual amplification of speech , 2008, Trends in Cognitive Sciences.

[53]  Nicolas Brunel,et al.  Encoding of Naturalistic Stimuli by Local Field Potential Spectra in Networks of Excitatory and Inhibitory Neurons , 2008, PLoS Comput. Biol..

[54]  M. Chait,et al.  Neural Correlates of Auditory Figure-Ground Segregation Based on Temporal Coherence , 2016, Cerebral cortex.

[55]  H. Scheich,et al.  A multilevel and cross-modal approach towards neuronal mechanisms of auditory streaming , 2008, Brain Research.

[56]  M. Sams,et al.  Time course of multisensory interactions during audiovisual speech perception in humans: a magnetoencephalographic study , 2004, Neuroscience Letters.

[57]  Stefano Panzeri,et al.  The Laminar and Temporal Structure of Stimulus Information in the Phase of Field Potentials of Auditory Cortex , 2011, The Journal of Neuroscience.

[58]  Christoph Kayser,et al.  Multisensory interactions in primate auditory cortex: fMRI and electrophysiology , 2009, Hearing Research.

[59]  Christoph Kayser,et al.  A Precluding But Not Ensuring Role of Entrained Low-Frequency Oscillations for Auditory Perception , 2012, The Journal of Neuroscience.