Integration of Visual Information in Auditory Cortex Promotes Auditory Scene Analysis through Multisensory Binding

Summary How and where in the brain audio-visual signals are bound to create multimodal objects remains unknown. One hypothesis is that temporal coherence between dynamic multisensory signals provides a mechanism for binding stimulus features across sensory modalities. Here, we report that when the luminance of a visual stimulus is temporally coherent with the amplitude fluctuations of one sound in a mixture, the representation of that sound is enhanced in auditory cortex. Critically, this enhancement extends to include both binding and non-binding features of the sound. We demonstrate that visual information conveyed from visual cortex via the phase of the local field potential is combined with auditory information within auditory cortex. These data provide evidence that early cross-sensory binding provides a bottom-up mechanism for the formation of cross-sensory objects and that one role for multisensory binding in auditory cortex is to support auditory scene analysis.

[1]  J. B. Levitt,et al.  Retinotopic organization of ferret suprasylvian cortex , 2006, Visual Neuroscience.

[2]  Rachel N. Denison,et al.  Temporal Structure and Complexity Affect Audio-Visual Correspondence Detection , 2013, Front. Psychology.

[3]  C. Schroeder,et al.  The Spectrotemporal Filter Mechanism of Auditory Selective Attention , 2013, Neuron.

[4]  Michael Okun,et al.  The Subthreshold Relation between Cortical Local Field Potential and Neuronal Firing Unveiled by Intracellular Recordings in Awake Rats , 2010, The Journal of Neuroscience.

[5]  Pablo Fuentealba,et al.  Cell Type-Specific Tuning of Hippocampal Interneuron Firing during Gamma Oscillations In Vivo , 2007, The Journal of Neuroscience.

[6]  Kerry M. M. Walker,et al.  Interdependent Encoding of Pitch, Timbre, and Spatial Location in Auditory Cortex , 2009, The Journal of Neuroscience.

[7]  R. Quian Quiroga,et al.  Unsupervised Spike Detection and Sorting with Wavelets and Superparamagnetic Clustering , 2004, Neural Computation.

[8]  Xiao-Jing Wang.  Neurophysiological and computational principles of cortical rhythms in cognition. , 2010, Physiological reviews.

[9]  Arne D. Ekstrom,et al.  Brain Oscillations Control Timing of Single-Neuron Activity in Humans , 2007, The Journal of Neuroscience.

[10]  Huriye Atilgan,et al.  The role of spectral cues in timbre discrimination by ferrets and humans. , 2015, The Journal of the Acoustical Society of America.

[11]  R. Freeman,et al.  Neurometabolic coupling in cerebral cortex reflects synaptic more than spiking activity , 2007, Nature Neuroscience.

[12]  Stefano Panzeri,et al.  Visual Enhancement of the Information Representation in Auditory Cortex , 2010, Current Biology.

[13]  N. Logothetis,et al.  Visual modulation of neurons in auditory cortex. , 2008, Cerebral cortex.

[14]  Henning Scheich,et al.  Neuronal activity in primate auditory cortex during the performance of audiovisual tasks , 2015, The European journal of neuroscience.

[15]  Christoph Kayser,et al.  Natural asynchronies in audiovisual communication signals regulate neuronal multisensory interactions in voice-sensitive cortex , 2014, Proceedings of the National Academy of Sciences.

[16]  Jennifer K. Bizley,et al.  Multiplexed and multivariate representations of sound identity during perceptual constancy , 2017, bioRxiv.

[17]  H. Scheich,et al.  A multilevel and cross-modal approach towards neuronal mechanisms of auditory streaming , 2008, Brain Research.

[18]  Gregory B. Cogan,et al.  Visual Input Enhances Selective Speech Envelope Tracking in Auditory Cortex at a “Cocktail Party” , 2013, The Journal of Neuroscience.

[19]  A. Puce,et al.  Neuronal oscillations and visual amplification of speech , 2008, Trends in Cognitive Sciences.

[20]  Guglielmo Foffani,et al.  PSTH-based classification of sensory stimuli using ensembles of single neurons , 2004, Journal of Neuroscience Methods.

[21]  J. Peelle,et al.  Prediction and constraint in audiovisual speech perception , 2015, Cortex.

[22]  Michael J. Crosse,et al.  Eye Can Hear Clearly Now: Inverse Effectiveness in Natural Audiovisual Speech Processing Relies on Long-Term Crossmodal Temporal Integration , 2016, The Journal of Neuroscience.

[23]  R. Desimone,et al.  Neural mechanisms of selective visual attention. , 1995, Annual review of neuroscience.

[24]  Adrian K. C. Lee,et al.  Defining Auditory-Visual Objects: Behavioral Tests and Physiological Mechanisms , 2016, Trends in Neurosciences.

[25]  D. P. Phillips,et al.  Representation of the cochlea in primary auditory cortex of the ferret (Mustela putorius) , 1986, Hearing Research.

[26]  John J. Foxe,et al.  Neuro-Oscillatory Phase Alignment Drives Speeded Multisensory Response Times: An Electro-Corticographic Investigation , 2015, The Journal of Neuroscience.

[27]  W. H. Sumby,et al.  Visual contribution to speech intelligibility in noise , 1954 .

[28]  H. Scheich,et al.  Multisensory processing via early cortical stages: Connections of the primary auditory cortical field with other sensory systems , 2006, Neuroscience.

[29]  Joost X. Maier,et al.  Multisensory Integration of Dynamic Faces and Voices in Rhesus Monkey Auditory Cortex , 2005 .

[30]  Michael J. Crosse,et al.  Congruent Visual Speech Enhances Cortical Entrainment to Continuous Auditory Speech in Noise-Free Conditions , 2015, The Journal of Neuroscience.

[31]  Asif A. Ghazanfar,et al.  The Natural Statistics of Audiovisual Speech , 2009, PLoS Comput. Biol..

[32]  Asif A Ghazanfar,et al.  Dynamic faces speed up the onset of auditory cortical spiking responses during vocal detection , 2013, Proceedings of the National Academy of Sciences.

[33]  C. Gray,et al.  Cellular Mechanisms Contributing to Response Variability of Cortical Neurons In Vivo , 1999, The Journal of Neuroscience.

[34]  Philipp Berens,et al.  CircStat: AMATLABToolbox for Circular Statistics , 2009, Journal of Statistical Software.

[35]  M. Sams,et al.  Time course of multisensory interactions during audiovisual speech perception in humans: a magnetoencephalographic study , 2004, Neuroscience Letters.

[36]  M. Malmierca,et al.  Effect of Auditory Cortex Deactivation on Stimulus-Specific Adaptation in the Medial Geniculate Body , 2011, The Journal of Neuroscience.

[37]  Stefano Panzeri,et al.  The Laminar and Temporal Structure of Stimulus Information in the Phase of Field Potentials of Auditory Cortex , 2011, The Journal of Neuroscience.

[38]  Thilo Womelsdorf,et al.  A Role of Phase-Resetting in Coordinating Large Scale Neural Networks During Attention and Goal-Directed Behavior , 2016, Front. Syst. Neurosci..

[39]  Gregory Hickok,et al.  An fMRI Study of Audiovisual Speech Perception Reveals Multisensory Interactions in Auditory Cortex , 2013, PloS one.

[40]  Nicolas Brunel,et al.  Encoding of Naturalistic Stimuli by Local Field Potential Spectra in Networks of Excitatory and Inhibitory Neurons , 2008, PLoS Comput. Biol..

[41]  Christoph Kayser,et al.  Multisensory interactions in primate auditory cortex: fMRI and electrophysiology , 2009, Hearing Research.

[42]  Christoph Kayser,et al.  A Precluding But Not Ensuring Role of Entrained Low-Frequency Oscillations for Auditory Perception , 2012, The Journal of Neuroscience.

[43]  Charles H. Brown,et al.  The Influence of Natural Scene Dynamics on Auditory Cortical Activity , 2010, The Journal of Neuroscience.

[44]  B. Postle,et al.  Top-down control of the phase of alpha-band oscillations as a mechanism for temporal prediction , 2015, Proceedings of the National Academy of Sciences.

[45]  Kerry M. M. Walker,et al.  Spectral timbre perception in ferrets: discrimination of artificial vowels under different listening conditions. , 2013, The Journal of the Acoustical Society of America.

[46]  C. Schroeder,et al.  Neuronal Oscillations and Multisensory Interaction in Primary Auditory Cortex , 2007, Neuron.

[47]  D. Poeppel,et al.  Auditory Cortex Tracks Both Auditory and Visual Stimulus Dynamics Using Low-Frequency Neuronal Phase Modulation , 2010, PLoS biology.

[48]  I. Nelken,et al.  Physiological and Anatomical Evidence for Multisensory Interactions in Auditory Cortex , 2006, Cerebral cortex.

[49]  P. Schyns,et al.  Cracking the Code of Oscillatory Activity , 2011, PLoS biology.

[50]  W. Klimesch,et al.  EEG alpha oscillations: The inhibition–timing hypothesis , 2007, Brain Research Reviews.

[51]  Jennifer K Bizley,et al.  Where are multisensory signals combined for perceptual decision-making? , 2016, Current Opinion in Neurobiology.

[52]  D. Poeppel,et al.  Phase Patterns of Neuronal Responses Reliably Discriminate Speech in Human Auditory Cortex , 2007, Neuron.

[53]  Kai Lu,et al.  Temporal coherence structure rapidly shapes neuronal interactions , 2017, Nature Communications.

[54]  M. Chait,et al.  Neural Correlates of Auditory Figure-Ground Segregation Based on Temporal Coherence , 2016, Cerebral cortex.

[55]  Adrian K. C. Lee,et al.  Auditory selective attention is enhanced by a task-irrelevant temporally coherent visual stimulus in human listeners , 2015, eLife.

[56]  S. Hughes,et al.  Temporal Framing of Thalamic Relay-Mode Firing by Phasic Inhibition during the Alpha Rhythm , 2009, Neuron.

[57]  Huriye Atilgan,et al.  Acute Inactivation of Primary Auditory Cortex Causes a Sound Localisation Deficit in Ferrets , 2017, PloS one.

[58]  Italo Masiello,et al.  Architecture and callosal connections of visual areas 17, 18, 19 and 21 in the ferret (Mustela putorius). , 2002, Cerebral cortex.

[59]  S. Shamma,et al.  Temporal Coherence in the Perceptual Organization and Cortical Representation of Auditory Scenes , 2009, Neuron.

[60]  James A. O'Sullivan,et al.  Evidence for Neural Computations of Temporal Coherence in an Auditory Scene and Their Enhancement during Active Listening , 2015, The Journal of Neuroscience.

[61]  R. Metherate,et al.  Spectral integration in primary auditory cortex: Laminar processing of afferent input, in vivo and in vitro , 2005, Neuroscience.

[62]  Gregor Thut,et al.  Lip movements entrain the observers’ low-frequency brain oscillations to facilitate speech intelligibility , 2016, eLife.