Effects of Sound Frequency on Audiovisual Integration: An Event-Related Potential Study

A combination of signals across modalities can facilitate sensory perception. The audiovisual facilitative effect strongly depends on the features of the stimulus. Here, we investigated how sound frequency, which is one of basic features of an auditory signal, modulates audiovisual integration. In this study, the task of the participant was to respond to a visual target stimulus by pressing a key while ignoring auditory stimuli, comprising of tones of different frequencies (0.5, 1, 2.5 and 5 kHz). A significant facilitation of reaction times was obtained following audiovisual stimulation, irrespective of whether the task-irrelevant sounds were low or high frequency. Using event-related potential (ERP), audiovisual integration was found over the occipital area for 0.5 kHz auditory stimuli from 190–210 ms, for 1 kHz stimuli from 170–200 ms, for 2.5 kHz stimuli from 140–200 ms, 5 kHz stimuli from 100–200 ms. These findings suggest that a higher frequency sound signal paired with visual stimuli might be early processed or integrated despite the auditory stimuli being task-irrelevant information. Furthermore, audiovisual integration in late latency (300–340 ms) ERPs with fronto-central topography was found for auditory stimuli of lower frequencies (0.5, 1 and 2.5 kHz). Our results confirmed that audiovisual integration is affected by the frequency of an auditory stimulus. Taken together, the neurophysiological results provide unique insight into how the brain processes a multisensory visual signal and auditory stimuli of different frequencies.

[1]  B. Moore An Introduction to the Psychology of Hearing: Sixth Edition , 2012 .

[2]  A. Starr,et al.  Auditory-evoked potentials to frequency increase and decrease of high- and low-frequency tones , 2009, Clinical Neurophysiology.

[3]  E. Macaluso Multisensory Processing in Sensory-Specific Cortical Areas , 2006, The Neuroscientist : a review journal bringing neurobiology, neurology and psychiatry.

[4]  William A. Yost,et al.  Fundamentals of Hearing: An Introduction: Fifth Edition , 2013 .

[5]  M. Woldorff,et al.  Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration? , 2006, Cerebral cortex.

[6]  John A. Molino Pure-tone equal-loudness contours for standard tones of different frequencies , 1971 .

[7]  Wei Liu,et al.  Human Cochlea: Anatomical Characteristics and their Relevance for Cochlear Implantation , 2012, Anatomical record.

[8]  A J Van Opstal,et al.  Auditory-visual interactions subserving goal-directed saccades in a complex scene. , 2002, Journal of neurophysiology.

[9]  Xiaoyu Tang,et al.  Effects of Auditory Stimuli in the Horizontal Plane on Audiovisual Integration: An Event-Related Potential Study , 2013, PloS one.

[10]  Birger Kollmeier,et al.  Spectral loudness summation takes place in the primary auditory cortex , 2011, Human brain mapping.

[11]  Jinglong Wu,et al.  Age‐related multisensory integration elicited by peripherally presented audiovisual stimuli , 2012, Neuroreport.

[12]  Thomas Elbert,et al.  Tonotopic organization of the human auditory cortex probed with frequency-modulated tones , 2004, Hearing Research.

[13]  M. Greicius,et al.  A cross‐modal system linking primary auditory and visual cortices: Evidence from intrinsic fMRI connectivity analysis , 2008, Human brain mapping.

[14]  J. Lewald,et al.  Cross-modal perceptual integration of spatially and temporally disparate auditory and visual stimuli. , 2003, Brain research. Cognitive brain research.

[15]  H Stanislaw,et al.  Calculation of signal detection theory measures , 1999, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[16]  Kestutis Kveraga,et al.  Multimodal access to verbal name codes , 2007, Perception & psychophysics.

[17]  J. Driver,et al.  Sound-Induced Enhancement of Low-Intensity Vision: Multisensory Influences on Human Sensory-Specific Cortices and Thalamic Bodies Relate to Perceptual Enhancement of Visual Detection Sensitivity , 2010, The Journal of Neuroscience.

[18]  Gideon Keren,et al.  A Handbook for data analysis in the behavioral sciences : methodological issues , 1993 .

[19]  Steven A. Hillyard,et al.  Effects of Spatial Congruity on Audio-Visual Multimodal Integration , 2005, Journal of Cognitive Neuroscience.

[20]  D. Bendor,et al.  The neuronal representation of pitch in primate auditory cortex , 2005, Nature.

[21]  J. Pickles An Introduction to the Physiology of Hearing , 1982 .

[22]  Tetsuo Touge,et al.  Multisensory Interactions Elicited by Audiovisual Stimuli Presented Peripherally in a Visual Attention Task: A Behavioral and Event-Related Potential Study in Humans , 2009, Journal of clinical neurophysiology : official publication of the American Electroencephalographic Society.

[23]  J. Pickles An Introduction to the Physiology of Hearing, Second Edition , 1988 .

[24]  Jozef J. Zwislocki,et al.  Cochlear mechanisms of frequency and intensity coding. I. The place code for pitch , 1997, Hearing Research.

[25]  S A Hillyard,et al.  An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. , 2002, Brain research. Cognitive brain research.

[26]  Christopher J. Plack,et al.  Pitch coding and pitch processing in the human brain , 2014, Hearing Research.

[27]  D. W. Robinson,et al.  Threshold of Hearing and Equal‐Loudness Relations for Pure Tones, and the Loudness Function , 1957 .

[28]  F. Bloom,et al.  Modulation of early sensory processing in human auditory cortex during auditory selective attention. , 1993, Proceedings of the National Academy of Sciences of the United States of America.

[29]  J. Vroomen,et al.  Sound enhances visual perception: cross-modal effects of auditory organization on vision. , 2000, Journal of experimental psychology. Human perception and performance.

[30]  Andrew J Oxenham,et al.  Correct tonotopic representation is necessary for complex pitch perception. , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[31]  John J. Foxe,et al.  The development of audiovisual multisensory integration across childhood and early adolescence: a high-density electrical mapping study. , 2011, Cerebral cortex.

[32]  Michael T. Lippert,et al.  Improvement of visual contrast detection by a simultaneous sound , 2007, Brain Research.

[33]  Jeff Miller,et al.  Timecourse of coactivation in bimodal divided attention , 1986, Perception & psychophysics.

[34]  David Poeppel,et al.  Visual speech speeds up the neural processing of auditory speech. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[35]  G E Loeb,et al.  Spatial cross-correlation , 1983, Biological Cybernetics.

[36]  V. Bruns,et al.  Cochlea in old world mice and rats (Muridae) , 1988, Journal of morphology.

[37]  Hans Colonius,et al.  Time-Window-of-Integration (TWIN) Model for Saccadic Reaction Time: Effect of Auditory Masker Level on Visual–Auditory Spatial Interaction in Elevation , 2009, Brain Topography.

[38]  J. Driver,et al.  Multisensory Interplay Reveals Crossmodal Influences on ‘Sensory-Specific’ Brain Regions, Neural Responses, and Judgments , 2008, Neuron.

[39]  Hans Colonius,et al.  On quantifying multisensory interaction effects in reaction time and detection rate , 2011, Psychological research.

[40]  Neil A. Macmillan,et al.  Signal detection theory as data analysis method and psychological decision model , 1993 .

[41]  Larry E. Roberts,et al.  Frequency organization of the 40-Hz auditory steady-state response in normal hearing and in tinnitus , 2006, NeuroImage.

[42]  A. Ghazanfar,et al.  Is neocortex essentially multisensory? , 2006, Trends in Cognitive Sciences.

[43]  M. Giard,et al.  Auditory-Visual Integration during Multimodal Object Recognition in Humans: A Behavioral and Electrophysiological Study , 1999, Journal of Cognitive Neuroscience.

[44]  Michael D. Rugg,et al.  Word and Nonword Repetition Within- and Across-Modality: An Event-Related Potential Study , 1995, Journal of Cognitive Neuroscience.

[45]  S. Uppenkamp,et al.  Neural Coding of Sound Intensity and Loudness in the Human Auditory System , 2012, Journal of the Association for Research in Otolaryngology.

[46]  H. Kennedy,et al.  Anatomical Evidence of Multimodal Integration in Primate Striate Cortex , 2002, The Journal of Neuroscience.

[47]  Joan López-Moliner,et al.  Sound-driven enhancement of vision: disentangling detection-level from decision-level contributions. , 2013, Journal of neurophysiology.

[48]  John J. Foxe,et al.  Multisensory interactions in early evoked brain activity follow the principle of inverse effectiveness , 2011, NeuroImage.

[49]  Sarah E Donohue,et al.  The Cross-Modal Spread of Attention Reveals Differential Constraints for the Temporal and Spatial Linking of Visual and Auditory Stimulus Events , 2011, The Journal of Neuroscience.

[50]  D. Barth,et al.  The spatiotemporal organization of auditory, visual, and auditory-visual evoked potentials in rat cortex , 1995, Brain Research.

[51]  Lee M. Miller,et al.  Tuning In to Sound: Frequency-Selective Attentional Filter in Human Primary Auditory Cortex , 2013, The Journal of Neuroscience.

[52]  John J. Foxe,et al.  Audio-visual multisensory integration in superior parietal lobule revealed by human intracranial recordings. , 2006, Journal of neurophysiology.

[53]  John Polich,et al.  Stimulus frequency and masking as determinants of P300 latency in event-related potentials from auditory stimuli , 1985, Biological Psychology.

[54]  S. Hillyard,et al.  Modulation of early auditory processing during selective listening to rapidly presented tones. , 1991, Electroencephalography and clinical neurophysiology.

[55]  D. Guthrie,et al.  Significance testing of difference potentials. , 1991, Psychophysiology.

[56]  Andrew J Oxenham,et al.  An autocorrelation model with place dependence to account for the effect of harmonic number on fundamental frequency discrimination. , 2005, The Journal of the Acoustical Society of America.

[57]  M. Tervaniemi,et al.  Pitch discrimination accuracy in musicians vs nonmusicians: an event-related potential and behavioral study , 2005, Experimental Brain Research.

[58]  John J. Foxe,et al.  Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. , 2002, Brain research. Cognitive brain research.

[59]  L. Busse,et al.  The spread of attention across modalities and space in a multisensory object. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[60]  B. Delgutte,et al.  Neural correlates of the pitch of complex tones. I. Pitch and pitch salience. , 1996, Journal of neurophysiology.

[61]  M. Müller,et al.  Frequency representation in the rat cochlea , 1991, Hearing Research.

[62]  Stefan Uppenkamp,et al.  Human auditory neuroimaging of intensity and loudness , 2014, Hearing Research.

[63]  John J. Foxe,et al.  Multisensory Representation of Frequency across Audition and Touch: High Density Electrical Mapping Reveals Early Sensory-Perceptual Coupling , 2012, The Journal of Neuroscience.

[64]  E. Owens,et al.  An Introduction to the Psychology of Hearing , 1997 .