Neural Oscillations Orchestrate Multisensory Processing

At any given moment, we receive input through our different sensory systems, and this information needs to be processed and integrated. Multisensory processing requires the coordinated activity of distinct cortical areas. Key mechanisms implicated in these processes include local neural oscillations and functional connectivity between distant cortical areas. Evidence is now emerging that neural oscillations in distinct frequency bands reflect different mechanisms of multisensory processing. Moreover, studies suggest that aberrant neural oscillations contribute to multisensory processing deficits in clinical populations, such as schizophrenia. In this article, we review recent literature on the neural mechanisms underlying multisensory processing, focusing on neural oscillations. We derive a framework that summarizes findings on (1) stimulus-driven multisensory processing, (2) the influence of top-down information on multisensory processing, and (3) the role of predictions for the formation of multisensory perception. We propose that different frequency band oscillations subserve complementary mechanisms of multisensory processing. These processes can act in parallel and are essential for multisensory processing.

[1]  G. Rees,et al.  Individual Differences in Alpha Frequency Drive Crossmodal Illusory Perception , 2015, Current Biology.

[2]  Arnaud Delorme,et al.  EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis , 2004, Journal of Neuroscience Methods.

[3]  Noriaki Kanayama,et al.  Top down influence on visuo-tactile interaction modulates neural oscillatory responses , 2012, NeuroImage.

[4]  John J. Foxe,et al.  Crossmodal binding through neural coherence: implications for multisensory processing , 2008, Trends in Neurosciences.

[5]  D. Senkowski,et al.  Early and late beta-band power reflect audiovisual perception in the McGurk illusion. , 2015, Journal of neurophysiology.

[6]  Christoph Kayser,et al.  Neurophysiological Correlates of the Rubber Hand Illusion in Late Evoked and Alpha/Beta Band Activity , 2017, Front. Hum. Neurosci..

[7]  B. McMurray,et al.  Can you hear me yet? An intracranial investigation of speech and non-speech audiovisual interactions in human cortex , 2016, Language, cognition and neuroscience.

[8]  J. Gross,et al.  Sounds Reset Rhythms of Visual Cortex and Corresponding Human Visual Perception , 2012, Current Biology.

[9]  S. Soto-Faraco,et al.  Theta oscillations reflect conflict processing in the perception of the McGurk illusion , 2018, The European journal of neuroscience.

[10]  D. Senkowski,et al.  Taking a Call Is Facilitated by the Multisensory Processing of Smartphone Vibrations, Sounds, and Flashes , 2014, PloS one.

[11]  D. Senkowski,et al.  Beta/Gamma Oscillations and Event-Related Potentials Indicate Aberrant Multisensory Processing in Schizophrenia , 2016, Front. Psychol..

[12]  Noriaki Kanayama,et al.  The role of gamma band oscillations and synchrony on rubber hand illusion and crossmodal integration , 2009, Brain and Cognition.

[13]  N. Weisz,et al.  Prestimulus beta power and phase synchrony influence the sound-induced flash illusion. , 2014, Cerebral cortex.

[14]  O. Araki,et al.  Phase reset affects auditory-visual simultaneity judgment , 2015, Cognitive Neurodynamics.

[15]  Shlomit Yuval-Greenberg,et al.  What You See Is Not (Always) What You Hear: Induced Gamma Band Responses Reflect Cross-Modal Interactions in Familiar Object Recognition , 2007, The Journal of Neuroscience.

[16]  David Melcher,et al.  Multiple oscillatory rhythms determine the temporal organization of perception , 2017, Proceedings of the National Academy of Sciences.

[17]  Shinsuke Shimojo,et al.  Sound-induced illusory flash perception: role of gamma band responses , 2002, Neuroreport.

[18]  T. Kircher,et al.  The EEG and fMRI signatures of neural integration: An investigation of meaningful gestures and corresponding speech , 2015, Neuropsychologia.

[19]  D. Senkowski,et al.  Individual Alpha Frequency Relates to the Sound-Induced Flash Illusion. , 2018, Multisensory research.

[20]  Arielle S. Keller,et al.  Characterizing the roles of alpha and theta oscillations in multisensory attention , 2017, Neuropsychologia.

[21]  O. Blanke,et al.  Alpha band oscillations correlate with illusory self‐location induced by virtual reality , 2011, The European journal of neuroscience.

[22]  Ole Jensen,et al.  A biologically plausible mechanism for neuronal coding organized by the phase of alpha oscillations , 2016, The European journal of neuroscience.

[23]  Luc H. Arnal,et al.  Transitions in neural oscillations reflect prediction errors generated in audiovisual speech , 2011, Nature Neuroscience.

[24]  Andreas K. Engel,et al.  Oscillatory signatures of crossmodal congruence effects: An EEG investigation employing a visuotactile pattern matching paradigm , 2015, NeuroImage.

[25]  Manuel R. Mercier,et al.  Auditory-driven phase reset in visual cortex: Human electrocorticography reveals mechanisms of early multisensory integration , 2013, NeuroImage.

[26]  Nathan Evans,et al.  Shared electrophysiology mechanisms of body ownership and motor imagery , 2013, NeuroImage.

[27]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[28]  Emanuele Porcu,et al.  Audio-visual synchrony and spatial attention enhance processing of dynamic visual stimulation independently and in parallel: A frequency-tagging study , 2017, NeuroImage.

[29]  C. Schroeder,et al.  Neuronal Oscillations and Multisensory Interaction in Primary Auditory Cortex , 2007, Neuron.

[30]  Luc H. Arnal,et al.  Cortical oscillations and sensory predictions , 2012, Trends in Cognitive Sciences.

[31]  Alexandre Gramfort,et al.  Encoding of event timing in the phase of neural oscillations , 2014, NeuroImage.

[32]  J. Obleser,et al.  How Bodies and Voices Interact in Early Emotion Perception , 2012, PloS one.

[33]  S. Shimojo,et al.  Illusions: What you see is what you hear , 2000, Nature.

[34]  Stefano Panzeri,et al.  Contributions of local speech encoding and functional connectivity to audio-visual speech perception , 2017, eLife.

[35]  Xiaorong Gao,et al.  EEG gamma-band activity during audiovisual speech comprehension in different noise environments , 2015, Cognitive Neurodynamics.

[36]  Semiha Aydin,et al.  GABA concentration in superior temporal sulcus predicts gamma power and perception in the sound-induced flash illusion , 2016, NeuroImage.

[37]  John J. Foxe,et al.  Neuro-Oscillatory Phase Alignment Drives Speeded Multisensory Response Times: An Electro-Corticographic Investigation , 2015, The Journal of Neuroscience.

[38]  N. Weisz,et al.  On the variability of the McGurk effect: audiovisual integration depends on prestimulus brain states. , 2012, Cerebral cortex.

[39]  Julian Keil,et al.  Visuotactile motion congruence enhances gamma-band activity in visual and somatosensory cortices , 2015, NeuroImage.

[40]  R. Oostenveld,et al.  Reduced Occipital Alpha Power Indexes Enhanced Excitability Rather than Improved Visual Perception , 2013, The Journal of Neuroscience.

[41]  Takashi Hanakawa,et al.  Neural oscillations in the temporal pole for a temporally congruent audio-visual speech detection task , 2016, Scientific Reports.

[42]  H. Coslett,et al.  Gone in a flash: manipulation of audiovisual temporal integration using transcranial magnetic stimulation , 2013, Front. Psychol..

[43]  Daniel Senkowski,et al.  Multisensory processing and oscillatory gamma responses: effects of spatial selective attention , 2005, Experimental Brain Research.

[44]  N. Logothetis,et al.  Visual modulation of neurons in auditory cortex. , 2008, Cerebral cortex.

[45]  Salvador Soto-Faraco,et al.  Speaker's Hand Gestures Modulate Speech Perception through Phase Resetting 1 of Ongoing Neural Oscillations 2 3 , 2022 .

[46]  J. Fadiman.  Illusions , 1999 .

[47]  Jess Hartcher-O’Brien,et al.  The Curious Incident of Attention in Multisensory Integration: Bottom-up vs. Top-down , 2016 .

[48]  Daniel Senkowski,et al.  Good times for multisensory integration: Effects of the precision of temporal synchrony as revealed by gamma-band oscillations , 2007, Neuropsychologia.

[49]  J. Driver,et al.  Attentional selection of location and modality in vision and touch modulates low-frequency activity in associated sensory cortices , 2012, Journal of neurophysiology.

[50]  Noriaki Kanayama,et al.  Crossmodal effect with rubber hand illusion and gamma-band activity. , 2007, Psychophysiology.

[51]  P. Roelfsema,et al.  Alpha and gamma oscillations characterize feedback and feedforward processing in monkey visual cortex , 2014, Proceedings of the National Academy of Sciences.

[52]  Christoph Kayser,et al.  Acoustic Noise Improves Visual Perception and Modulates Occipital Oscillatory States , 2014, Journal of Cognitive Neuroscience.

[53]  Matthias M. Müller,et al.  Audio-visual synchrony and spatial attention enhance processing of dynamic visual stimulation independently and in parallel: a frequency-tagging study , 2017, bioRxiv.

[54]  Virginie van Wassenhove,et al.  Prestimulus Alpha Oscillations and the Temporal Sequencing of Audiovisual Events , 2017, Journal of Cognitive Neuroscience.

[55]  Karl J. Friston,et al.  Task relevance modulates the behavioural and neural effects of sensory predictions , 2017, PLoS biology.

[56]  A. Banerjee,et al.  Large Scale Functional Brain Networks Underlying Temporal Integration of Audio-Visual Speech Perception: An EEG Study , 2016, Front. Psychol..

[57]  P. Fries.  Rhythms for Cognition: Communication through Coherence , 2015, Neuron.

[58]  P. Fries,et al.  Diverse Phase Relations among Neuronal Rhythms and Their Potential Function , 2016, Trends in Neurosciences.

[59]  J. Obleser,et al.  Frequency modulation entrains slow neural oscillations and optimizes human listening behavior , 2012, Proceedings of the National Academy of Sciences.

[60]  Chrysa D. Lithari,et al.  Prestimulus oscillatory alpha power and connectivity patterns predispose perceptual integration of an audio and a tactile stimulus , 2015, Human brain mapping.

[61]  A. Engel,et al.  Spectral fingerprints of large-scale neuronal interactions , 2012, Nature Reviews Neuroscience.

[62]  M. Murray,et al.  Multisensory Integration: Flexible Use of General Operations , 2014, Neuron.

[63]  H. Kennedy,et al.  Alpha-Beta and Gamma Rhythms Subserve Feedback and Feedforward Influences among Human Visual Cortical Areas , 2016, Neuron.

[64]  O. Bertrand,et al.  Oscillatory gamma activity in humans and its role in object representation , 1999, Trends in Cognitive Sciences.

[65]  David Poeppel,et al.  Cortical oscillations and speech processing: emerging computational principles and operations , 2012, Nature Neuroscience.

[66]  Brigitte Röder,et al.  Semantic confusion regarding the development of multisensory integration: a practical solution , 2010, The European journal of neuroscience.

[67]  P. Mitra,et al.  Analysis of dynamic brain imaging data. , 1998, Biophysical journal.

[68]  Joachim Lange,et al.  Audio–visual congruency alters power and coherence of oscillatory activity within and between cortical areas , 2013, NeuroImage.

[69]  Michael S. Beauchamp,et al.  Electrocorticography Links Human Temporoparietal Junction to Visual Perception , 2012, Nature Neuroscience.

[70]  E. Macaluso,et al.  Audiovisual integration as conflict resolution: The conflict of the McGurk illusion , 2017, Human brain mapping.

[71]  Mark T. Wallace,et al.  Rhythmic Modulation of Entrained Auditory Oscillations by Visual Inputs , 2017, Brain Topography.

[72]  J. Peelle,et al.  Prediction and constraint in audiovisual speech perception , 2015, Cortex.

[73]  Ankoor S. Shah,et al.  An oscillatory hierarchy controlling neuronal excitability and stimulus processing in the auditory cortex. , 2005, Journal of neurophysiology.

[74]  Jeff Miller,et al.  Timecourse of coactivation in bimodal divided attention , 1986, Perception & psychophysics.

[75]  Julian Keil,et al.  The role of alpha oscillations for illusory perception , 2014, Behavioural Brain Research.

[76]  E. Halgren,et al.  Intracranial Cortical Responses during Visual–Tactile Integration in Humans , 2014, The Journal of Neuroscience.

[77]  Anne-Lise Giraud,et al.  The contribution of frequency-specific activity to hierarchical information processing in the human auditory cortex , 2014, Nature Communications.

[78]  Christoph Kayser,et al.  Sounds facilitate visual motion discrimination via the enhancement of late occipital visual representations , 2017, NeuroImage.

[79]  John J. Foxe,et al.  Recalibration of the Multisensory Temporal Window of Integration Results from Changing Task Demands , 2013, PloS one.

[80]  A. Engel,et al.  Beta-band oscillations—signalling the status quo? , 2010, Current Opinion in Neurobiology.

[81]  Xiao-Jing Wang.  Neurophysiological and computational principles of cortical rhythms in cognition. , 2010, Physiological reviews.

[82]  Xiting Huang,et al.  Pre-stimulus beta and gamma oscillatory power predicts perceived audiovisual simultaneity. , 2016, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[83]  Robert Oostenveld,et al.  Perception of the touch-induced visual double-flash illusion correlates with changes of rhythmic neuronal activity in human visual and somatosensory areas , 2011, NeuroImage.

[84]  D. Senkowski,et al.  Reduced frontal theta oscillations indicate altered crossmodal prediction error processing in schizophrenia. , 2016, Journal of neurophysiology.

[85]  S. Jones.  When brain rhythms aren't ‘rhythmic’: implication for their mechanisms and meaning , 2016, Current Opinion in Neurobiology.

[86]  D. Senkowski,et al.  Alpha-Band Oscillations Reflect Altered Multisensory Processing of the McGurk Illusion in Schizophrenia , 2016, Front. Hum. Neurosci..

[87]  Jason B. Mattingley,et al.  Parietal disruption alters audiovisual binding in the sound-induced flash illusion , 2012, NeuroImage.

[88]  I. Nelken,et al.  Transient Induced Gamma-Band Response in EEG as a Manifestation of Miniature Saccades , 2008, Neuron.

[89]  M. Frank,et al.  Frontal theta as a mechanism for cognitive control , 2014, Trends in Cognitive Sciences.

[90]  H. Mizuhara,et al.  Top-down and bottom-up attention cause the ventriloquism effect with distinct electroencephalography modulations , 2016, Neuroreport.

[91]  John J. Foxe,et al.  Intersensory selective attention and temporal orienting operate in parallel and are instantiated in spatially distinct sensory and motor cortices , 2015, Human brain mapping.

[92]  W. El-Deredy,et al.  Pre-stimulus alpha oscillations over somatosensory cortex predict tactile misperceptions , 2017, Neuropsychologia.

[93]  Rajesh P. N. Rao,et al.  Predictive Coding , 2019, A Blueprint for the Hard Problem of Consciousness.

[94]  Ulrich Pomper,et al.  Distinct patterns of local oscillatory activity and functional connectivity underlie intersensory attention and temporal prediction , 2016, Cortex.

[95]  Christoph Kayser,et al.  Oscillatory mechanisms underlying the enhancement of visual motion perception by multisensory congruency , 2014, Neuropsychologia.

[96]  Andreas K. Engel,et al.  Attention Modulates Visual-Tactile Interaction in Spatial Pattern Matching , 2014, PloS one.

[97]  Dean R. Freestone,et al.  Evidence for Enhanced Multisensory Facilitation with Stimulus Relevance: An Electrophysiological Investigation , 2013, PloS one.

[98]  Andreas K. Engel,et al.  Oscillatory Synchronization in Large-Scale Cortical Networks Predicts Perception , 2011, Neuron.

[99]  O. Jensen,et al.  Shaping Functional Architecture by Oscillatory Alpha Activity: Gating by Inhibition , 2010, Front. Hum. Neurosci..

[100]  K. Hiraki,et al.  Cortical EEG components that reflect inverse effectiveness during visuotactile integration processing , 2015, Brain Research.

[101]  B. Stein,et al.  The Merging of the Senses , 1993 .

[102]  T. Sejnowski,et al.  Early Cross-Modal Interactions in Auditory and Visual Cortex Underlie a Sound-Induced Visual Illusion , 2007, The Journal of Neuroscience.

[103]  Andreas K. Engel,et al.  Noise alters beta-band activity in superior temporal cortex during audiovisual speech processing , 2013, NeuroImage.