Interacting parallel pathways associate sounds with visual identity in auditory cortices

Spatial and non-spatial information of sound events is presumably processed in parallel auditory cortex (AC) "what" and "where" streams, which are modulated by inputs from the respective visual-cortex subsystems. How these parallel processes are integrated to perceptual objects that remain stable across time and the source agent's movements is unknown. We recorded magneto- and electroencephalography (MEG/EEG) data while subjects viewed animated video clips featuring two audiovisual objects, a black cat and a gray cat. Adaptor-probe events were either linked to the same object (the black cat meowed twice in a row in the same location) or included a visually conveyed identity change (the black and then the gray cat meowed with identical voices in the same location). In addition to effects in visual (including fusiform, middle temporal or MT areas) and frontoparietal association areas, the visually conveyed object-identity change was associated with a release from adaptation of early (50-150ms) activity in posterior ACs, spreading to left anterior ACs at 250-450ms in our combined MEG/EEG source estimates. Repetition of events belonging to the same object resulted in increased theta-band (4-8Hz) synchronization within the "what" and "where" pathways (e.g., between anterior AC and fusiform areas). In contrast, the visually conveyed identity changes resulted in distributed synchronization at higher frequencies (alpha and beta bands, 8-32Hz) across different auditory, visual, and association areas. The results suggest that sound events become initially linked to perceptual objects in posterior AC, followed by modulations of representations in anterior AC. Hierarchical what and where pathways seem to operate in parallel after repeating audiovisual associations, whereas the resetting of such associations engages a distributed network across auditory, visual, and multisensory areas.

[1]  B Kollmeier,et al.  Dichotic Pitch activates Pitch Processing Centre in Heschl's Gyrus , 2009, NeuroImage.

[2]  R. Zatorre,et al.  Adaptation to speaker's voice in right anterior temporal lobe , 2003, Neuroreport.

[3]  W. Singer,et al.  Visuomotor integration is associated with zero time-lag synchronization among cortical areas , 1997, Nature.

[4]  D. Vaitl,et al.  Utilizing the ventriloquism-effect to investigate audio-visual binding , 2007, Neuropsychologia.

[5]  J. Rauschecker,et al.  Functional Specialization in Rhesus Monkey Auditory Cortex , 2001, Science.

[6]  Lei Ding,et al.  Simultaneous EEG and MEG source reconstruction in sparse electromagnetic source imaging , 2013, Human brain mapping.

[7]  S. Ochs.  Integrative Activity of the Brain: An Interdisciplinary Approach , 1968 .

[8]  Shin'ya Nishida,et al.  Visual search for a target changing in synchrony with an auditory signal , 2006, Proceedings of the Royal Society B: Biological Sciences.

[9]  Siegel Markus,et al.  Oscillatory synchronization in large-scale cortical networks predicts perception , 2011 .

[10]  E. Bullmore,et al.  Activation of auditory cortex during silent lipreading. , 1997, Science.

[11]  J. Rauschecker,et al.  Mechanisms and streams for processing of "what" and "where" in auditory cortex. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[12]  W. Singer,et al.  Dynamic predictions: Oscillations and synchrony in top–down processing , 2001, Nature Reviews Neuroscience.

[13]  R. Ilmoniemi,et al.  Magnetoencephalography-theory, instrumentation, and applications to noninvasive studies of the working human brain , 1993 .

[14]  Y. Cohen,et al.  The what, where and how of auditory-object perception , 2013, Nature Reviews Neuroscience.

[15]  U. Noppeney,et al.  Superadditive responses in superior temporal sulcus predict audiovisual benefits in object categorization. , 2010, Cerebral cortex.

[16]  Seppo P. Ahlfors,et al.  Direction of magnetoencephalography sources associated with feedback and feedforward contributions in a visual object recognition task , 2015, Neuroscience Letters.

[17]  K. Rockland,et al.  Laminar origins and terminations of cortical connections of the occipital lobe in the rhesus monkey , 1979, Brain Research.

[18]  R. Patterson,et al.  The Processing of Temporal Pitch and Melody Information in Auditory Cortex , 2002, Neuron.

[19]  Anders M. Dale,et al.  Improved Localization of Cortical Activity By Combining EEG and MEG with MRI Cortical Surface Reconstruction , 2002 .

[20]  T. Hackett,et al.  Multisensory convergence in auditory cortex, I. Cortical connections of the caudal superior temporal plane in macaque monkeys , 2007, The Journal of comparative neurology.

[21]  宁北芳,et al.  疟原虫var基因转换速率变化导致抗原变异[英]／Paul H, Robert P, Christodoulou Z, et al//Proc Natl Acad Sci U S A , 2005 .

[22]  Roger B. H. Tootell,et al.  The advantage of combining MEG and EEG: Comparison to fMRI in focally stimulated visual cortex , 2007, NeuroImage.

[23]  I. Nelken,et al.  Physiological and Anatomical Evidence for Multisensory Interactions in Auditory Cortex , 2006, Cerebral cortex.

[24]  M. Sams,et al.  Adaptation of neuromagnetic N1 responses to phonetic stimuli by visual speech in humans , 2004, Neuroreport.

[25]  Tomas Knapen,et al.  Interregional alpha-band synchrony supports temporal cross-modal integration , 2014, NeuroImage.

[26]  M. Sams,et al.  Primary auditory cortex activation by visual speech: an fMRI study at 3 T , 2005, Neuroreport.

[27]  Stephen R. Arnott,et al.  The Functional Organization of Auditory Working Memory as Revealed by fMRI , 2005, Journal of Cognitive Neuroscience.

[28]  H Petsche,et al.  Synchronization between temporal and parietal cortex during multimodal object processing in man. , 1999, Cerebral cortex.

[29]  M. Mishkin,et al.  Dual streams of auditory afferents target multiple domains in the primate prefrontal cortex , 1999, Nature Neuroscience.

[30]  S. Nishida,et al.  A common perceptual temporal limit of binding synchronous inputs across different sensory attributes and modalities , 2010, Proceedings of the Royal Society B: Biological Sciences.

[31]  Andreas Kleinschmidt,et al.  Interaction of Face and Voice Areas during Speaker Recognition , 2005, Journal of Cognitive Neuroscience.

[32]  Mikko Sams,et al.  Formant transition-specific adaptation by lipreading of left auditory cortex N1m , 2008, Neuroreport.

[33]  Edward Awh,et al.  The Capacity of Audiovisual Integration Is Limited to One Item , 2013, Psychological science.

[34]  F. Lin,et al.  Onset timing of cross‐sensory activations and multisensory interactions in auditory and visual sensory cortices , 2010, The European journal of neuroscience.

[35]  J. Rauschecker,et al.  Phoneme and word recognition in the auditory ventral stream , 2012, Proceedings of the National Academy of Sciences.

[36]  Robert Oostenveld,et al.  An improved index of phase-synchronization for electrophysiological data in the presence of volume-conduction, noise and sample-size bias , 2011, NeuroImage.

[37]  N. Logothetis,et al.  A voice region in the monkey brain , 2008, Nature Neuroscience.

[38]  Claude Alain,et al.  Working memory load modulates the auditory “What” and “Where” neural networks , 2011, NeuroImage.

[39]  J. Kaiser,et al.  Human gamma-frequency oscillations associated with attention and memory , 2007, Trends in Neurosciences.

[40]  J. Rauschecker,et al.  A PET study of human auditory spatial processing , 1999, Neuroscience Letters.

[41]  John J. Foxe,et al.  Crossmodal binding through neural coherence: implications for multisensory processing , 2008, Trends in Neurosciences.

[42]  Karl J. Friston,et al.  MEG and EEG data fusion: Simultaneous localisation of face-evoked responses , 2009, NeuroImage.

[43]  K. G. Munhall,et al.  Audiovisual Integration of Speech in a Bistable Illusion , 2009, Current Biology.

[44]  F. Lin,et al.  Primary and multisensory cortical activity is correlated with audiovisual percepts , 2009, Human brain mapping.

[45]  J. Konorski.  Integrative activity of the brain , 1967 .

[46]  D. Senkowski,et al.  The multifaceted interplay between attention and multisensory integration , 2010, Trends in Cognitive Sciences.

[47]  S. Lomber,et al.  Double dissociation of 'what' and 'where' processing in auditory cortex , 2008, Nature Neuroscience.

[48]  H. Scheich,et al.  Multisensory processing via early cortical stages: Connections of the primary auditory cortical field with other sensory systems , 2006, Neuroscience.

[49]  E Mouchlianitis,et al.  MEG and EEG data fusion: simultaneous localisation of face-evoked responses , 2009, NeuroImage.

[50]  Mikko Sams,et al.  Attention-driven auditory cortex short-term plasticity helps segregate relevant sounds from noise , 2011, Proceedings of the National Academy of Sciences.

[51]  E. Maris,et al.  Prior Expectation Mediates Neural Adaptation to Repeated Sounds in the Auditory Cortex: An MEG Study , 2011, The Journal of Neuroscience.

[52]  Essa Yacoub,et al.  Encoding of Natural Sounds at Multiple Spectral and Temporal Resolutions in the Human Auditory Cortex , 2014, PLoS Comput. Biol..

[53]  A. Dale,et al.  Distributed current estimates using cortical orientation constraints , 2006, Human brain mapping.

[54]  Jon Driver,et al.  Audiovisual synchrony enhances BOLD responses in a brain network including multisensory STS while also enhancing target‐detection performance for both modalities , 2011, Human brain mapping.

[55]  J. Rauschecker,et al.  Evidence for distinct human auditory cortex regions for sound location versus identity processing , 2013, Nature Communications.

[56]  John J. Foxe,et al.  The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. , 2002, Brain research. Cognitive brain research.

[57]  David Alais,et al.  Multisensory Congruency as a Mechanism for Attentional Control over Perceptual Selection , 2009, The Journal of Neuroscience.

[58]  J. Lewald,et al.  Allocentric or Craniocentric Representation of Acoustic Space: An Electrotomography Study Using Mismatch Negativity , 2012, PloS one.

[59]  S. Hillyard,et al.  Neural Basis of the Ventriloquist Illusion , 2007, Current Biology.

[60]  R. Campbell,et al.  Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex , 2000, Current Biology.

[61]  K. Nakayama,et al.  RESPONSE PROPERTIES OF THE HUMAN FUSIFORM FACE AREA , 2000, Cognitive neuropsychology.

[62]  G. Recanzone,et al.  Serial and parallel processing in the primate auditory cortex revisited , 2010, Behavioural Brain Research.

[63]  J. Thiran,et al.  Distinct Pathways Involved in Sound Recognition and Localization: A Human fMRI Study , 2000, NeuroImage.

[64]  Risto Näätänen,et al.  Frequency Change Detection in Human Auditory Cortex , 1999, Journal of Computational Neuroscience.

[65]  L. Guarente,et al.  Functional Specialization in Rhesus Monkey Auditory Cortex , 2001 .

[66]  A. Dale,et al.  Human posterior auditory cortex gates novel sounds to consciousness. , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[67]  Matti S. Hämäläinen,et al.  How anatomical asymmetry of human auditory cortex can lead to a rightward bias in auditory evoked fields , 2013, NeuroImage.

[68]  T. Griffiths,et al.  The planum temporale as a computational hub , 2002, Trends in Neurosciences.

[69]  David Poeppel,et al.  Visual speech speeds up the neural processing of auditory speech. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[70]  A. Dale,et al.  High‐resolution intersubject averaging and a coordinate system for the cortical surface , 1999, Human brain mapping.

[71]  Jyrki Ahveninen,et al.  Auditory Conflict Resolution Correlates with Medial–Lateral Frontal Theta/Alpha Phase Synchrony , 2014, PloS one.

[72]  Mikko Sams,et al.  Enhanced neural synchrony between left auditory and premotor cortex is associated with successful phonetic categorization , 2013, Front. Psychol..

[73]  C. Grady,et al.  “What” and “where” in the human auditory system , 2001, Proceedings of the National Academy of Sciences of the United States of America.

[74]  M. Bornstein,et al.  Attention and memory. , 1989, Pediatric annals.

[75]  G. Ermentrout,et al.  Gamma rhythms and beta rhythms have different synchronization properties. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[76]  Aina Puce,et al.  Category-sensitive excitatory and inhibitory processes in human extrastriate cortex. , 2002, Journal of neurophysiology.

[77]  B. Argall,et al.  Integration of Auditory and Visual Information about Objects in Superior Temporal Sulcus , 2004, Neuron.

[78]  Riitta Hari,et al.  Audiovisual Integration of Letters in the Human Brain , 2000, Neuron.

[79]  G. Recanzone.  Rapidly induced auditory plasticity: the ventriloquism aftereffect. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[80]  Jean-Philippe Thiran,et al.  What and Where in human audition: selective deficits following focal hemispheric lesions , 2002, Experimental Brain Research.

[81]  Micah M. Murray,et al.  Top-down control and early multisensory processes: chicken vs. egg , 2015, Front. Integr. Neurosci..

[82]  Jochen Kaiser,et al.  Processing of location and pattern changes of natural sounds in the human auditory cortex , 2007, NeuroImage.

[83]  L. Ada.  Working memory load modulates the auditory "what" and "where" neural networks , 2011 .

[84]  Anders M. Dale,et al.  An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest , 2006, NeuroImage.

[85]  John W Belliveau,et al.  Monte Carlo simulation studies of EEG and MEG localization accuracy , 2002, Human brain mapping.

[86]  J. Rauschecker,et al.  Maps and streams in the auditory cortex: nonhuman primates illuminate human speech processing , 2009, Nature Neuroscience.

[87]  C. Avendano,et al.  The CIPIC HRTF database , 2001, Proceedings of the 2001 IEEE Workshop on the Applications of Signal Processing to Audio and Acoustics (Cat. No.01TH8575).

[88]  R. Hari,et al.  Seeing speech: visual information from lip movements modifies activity in the human auditory cortex , 1991, Neuroscience Letters.

[89]  Leslie G. Ungerleider,et al.  ‘What’ and ‘where’ in the human brain , 1994, Current Opinion in Neurobiology.

[90]  E. Halgren,et al.  Top-down facilitation of visual recognition. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[91]  Lawrence M. Ward,et al.  Asynchrony from synchrony: long-range gamma-band neural synchrony accompanies perception of audiovisual speech asynchrony , 2008, Experimental Brain Research.

[92]  L. Cauller.  Layer I of primary sensory neocortex: where top-down converges upon bottom-up , 1995, Behavioural Brain Research.

[93]  Doug J. K. Barrett,et al.  Response preferences for “what” and “where” in human non-primary auditory cortex , 2006, NeuroImage.

[94]  Martin Luessi,et al.  MNE software for processing MEG and EEG data , 2014, NeuroImage.

[95]  L. Busse,et al.  The spread of attention across modalities and space in a multisensory object. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[96]  R. Campbell,et al.  Audiovisual Integration of Speech Falters under High Attention Demands , 2005, Current Biology.

[97]  Luc H. Arnal,et al.  Transitions in neural oscillations reflect prediction errors generated in audiovisual speech , 2011, Nature Neuroscience.

[98]  Joost X. Maier,et al.  Multisensory Integration of Dynamic Faces and Voices in Rhesus Monkey Auditory Cortex , 2005 .

[99]  B. Shinn-Cunningham,et al.  Task-modulated “what” and “where” pathways in human auditory cortex , 2006, Proceedings of the National Academy of Sciences.

[100]  S. Taulu,et al.  Applications of the signal space separation method , 2005, IEEE Transactions on Signal Processing.

[101]  D. Poeppel,et al.  Auditory Cortex Tracks Both Auditory and Visual Stimulus Dynamics Using Low-Frequency Neuronal Phase Modulation , 2010, PLoS biology.

[102]  Dominique L. Pritchett,et al.  Neural Correlates of Tactile Detection: A Combined Magnetoencephalography and Biophysically Based Computational Modeling Study , 2007, The Journal of Neuroscience.

[103]  A. Puce,et al.  Neuronal oscillations and visual amplification of speech , 2008, Trends in Cognitive Sciences.

[104]  G B Ermentrout,et al.  Fine structure of neural spiking and synchronization in the presence of conduction delays. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[105]  R. Oostenveld,et al.  Nonparametric statistical testing of EEG- and MEG-data , 2007, Journal of Neuroscience Methods.

[106]  M. Andermann,et al.  Short-term plasticity as a neural mechanism supporting memory and attentional functions , 2011, Brain Research.

[107]  Joshua I. Breier,et al.  Functional neuroimaging with MEG: Normative language profiles , 2006, NeuroImage.

[108]  N. Logothetis,et al.  Auditory and Visual Modulation of Temporal Lobe Neurons in Voice-Sensitive and Association Cortices , 2014, The Journal of Neuroscience.

[109]  Bradley S. Peterson,et al.  Contextual control of audiovisual integration in low‐level sensory cortices , 2013, Human brain mapping.

[110]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[111]  J. Lewald.  Rapid adaptation to auditory-visual spatial disparity. , 2002, Learning & memory.

[112]  N. Logothetis,et al.  Integration of Touch and Sound in Auditory Cortex , 2005, Neuron.