Top-down control and early multisensory processes: chicken vs. egg

Traditional views contend that behaviorally-relevant multisensory interactions occur relatively late during stimulus processing and subsequently to influences of (top-down) attentional control. In contrast, work from the last 15 years shows that information from different senses is integrated in the brain also during the initial 100 ms after stimulus onset and within low-level cortices. Critically, many of these early-latency multisensory interactions (hereafter eMSI) directly impact behavior. The prevalence of eMSI substantially advances our understanding of how unified perception and goal-related behavior emerge. However, it also raises important questions about the dependency of the eMSI on top-down, goal-based attentional control mechanisms that bias information processing toward task-relevant objects (hereafter top-down control). To date, this dependency remains controversial, because eMSI can occur independently of top-down control, making it plausible for (some) multisensory processes to directly shape perception and behavior. In other words, the former is not necessary for these early effects to occur and to link them with perception (see Figure ​Figure1A).1A). This issue epitomizes the fundamental question regarding direct links between sensation, perception, and behavior (direct perception), and also extends it in a crucial way to incorporate the multisensory nature of everyday experience. At the same time, the emerging framework must strive to also incorporate the variety of higher-order control mechanisms that likely influence multisensory stimulus responses but which are not based on task-relevance. This article presents a critical perspective about the importance of top-down control for eMSI: In other words, who is controlling whom? Figure 1 (A) Depiction of manners in which top-down attentional control and bottom-up multisensory processes may influence direct perception in multisensory contexts. In this model, the bottom-up multisensory processes that occur early in time (eMSI; beige box) ...

[1]  M. Murray,et al.  Looming Signals Reveal Synergistic Principles of Multisensory Integration , 2012, The Journal of Neuroscience.

[2]  D. Barth,et al.  The spatiotemporal organization of auditory, visual, and auditory-visual evoked potentials in rat cortex , 1995, Brain Research.

[3]  Jan Theeuwes,et al.  Early multisensory interactions affect the competition among multiple visual objects , 2011, NeuroImage.

[4]  Marty G. Woldorff,et al.  Selective Attention and Multisensory Integration: Multiple Phases of Effects on the Evoked Brain Activity , 2005, Journal of Cognitive Neuroscience.

[5]  K. Grill-Spector,et al.  Repetition and the brain: neural models of stimulus-specific effects , 2006, Trends in Cognitive Sciences.

[6]  John J. Foxe,et al.  Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. , 2002, Brain research. Cognitive brain research.

[7]  S A Hillyard,et al.  An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. , 2002, Brain research. Cognitive brain research.

[8]  R. Reilly,et al.  Neural Mass Model of Human Multisensory Integration , 2006, 2006 International Conference of the IEEE Engineering in Medicine and Biology Society.

[9]  Sabine Kastner,et al.  The Oxford Handbook of Attention , 2014 .

[10]  Gregor Thut,et al.  Preperceptual and Stimulus-Selective Enhancement of Low-Level Human Visual Cortex Excitability by Sounds , 2009, Current Biology.

[11]  N. Logothetis,et al.  Visual modulation of neurons in auditory cortex. , 2008, Cerebral cortex.

[12]  F. Lin,et al.  Onset timing of cross‐sensory activations and multisensory interactions in auditory and visual sensory cortices , 2010, The European journal of neuroscience.

[13]  C. Summerfield,et al.  Expectation (and attention) in visual cognition , 2009, Trends in Cognitive Sciences.

[14]  Brigitte Röder,et al.  Semantic confusion regarding the development of multisensory integration: a practical solution , 2010, The European journal of neuroscience.

[15]  N. Bruneau,et al.  Cross-modal processing of auditory–visual stimuli in a no-task paradigm: A topographic event-related potential study , 2008, Clinical Neurophysiology.

[16]  Christoph M. Michel,et al.  Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging , 2004, NeuroImage.

[17]  J. Vroomen,et al.  Deficient multisensory integration in schizophrenia: An event-related potential study , 2013, Schizophrenia Research.

[18]  Mark W. Greenlee,et al.  Modality shift effects mimic multisensory interactions: an event-related potential study , 2007, Experimental Brain Research.

[19]  L. Merabet,et al.  Occipital Transcranial Magnetic Stimulation Has Opposing Effects on Visual and Auditory Stimulus Detection: Implications for Multisensory Interactions , 2007, The Journal of Neuroscience.

[20]  C. Schroeder,et al.  Audiovisual Integration in Nonhuman Primates: A Window into the Anatomy and Physiology of Cognition , 2011 .

[21]  J. Pernier,et al.  Early auditory-visual interactions in human cortex during nonredundant target identification. , 2002, Brain research. Cognitive brain research.

[22]  V. Lamme,et al.  The distinct modes of vision offered by feedforward and recurrent processing , 2000, Trends in Neurosciences.

[23]  R. Campbell,et al.  Audiovisual Integration of Speech Falters under High Attention Demands , 2005, Current Biology.

[24]  Gregor Thut,et al.  The Contributions of Sensory Dominance and Attentional Bias to Cross-modal Enhancement of Visual Cortex Excitability , 2013, Journal of Cognitive Neuroscience.

[25]  Marcelo A. Montemurro,et al.  Spike-Phase Coding Boosts and Stabilizes Information Carried by Spatial and Temporal Spike Patterns , 2009, Neuron.

[26]  M. Murray,et al.  Multisensory Integration: Flexible Use of General Operations , 2014, Neuron.

[27]  J. Theeuwes,et al.  Attention and the multiple stages of multisensory integration: A review of audiovisual studies. , 2010, Acta psychologica.

[28]  John J. Foxe,et al.  Multisensory visual-auditory object recognition in humans: a high-density electrical mapping study. , 2004, Cerebral cortex.

[29]  M. Wallace,et al.  Learning to Associate Auditory and Visual Stimuli: Behavioral and Neural Mechanisms , 2015, Brain Topography.

[30]  M. Woldorff,et al.  Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration? , 2006, Cerebral cortex.

[31]  Denis Brunet,et al.  Topographic ERP Analyses: A Step-by-Step Tutorial Review , 2008, Brain Topography.

[32]  Martin Eimer,et al.  Multisensory enhancement of attentional capture in visual search , 2011, Psychonomic bulletin & review.

[33]  Micah M. Murray,et al.  Multisensory context portends object memory , 2014, Current Biology.

[34]  Salvador Soto-Faraco,et al.  Attention to touch weakens audiovisual speech integration , 2007, Experimental Brain Research.

[35]  Brigitte Röder,et al.  A new method for detecting interactions between the senses in event-related potentials , 2006, Brain Research.

[36]  R. Desimone,et al.  Neural mechanisms of spatial selective attention in areas V1, V2, and V4 of macaque visual cortex. , 1997, Journal of neurophysiology.

[37]  M. Giard,et al.  Auditory-Visual Integration during Multimodal Object Recognition in Humans: A Behavioral and Electrophysiological Study , 1999, Journal of Cognitive Neuroscience.

[38]  G. Karmos,et al.  Entrainment of Neuronal Oscillations as a Mechanism of Attentional Selection , 2008, Science.

[39]  Shlomit Yuval-Greenberg,et al.  What You See Is Not (Always) What You Hear: Induced Gamma Band Responses Reflect Cross-Modal Interactions in Familiar Object Recognition , 2007, The Journal of Neuroscience.

[40]  C. Schroeder,et al.  Neuronal mechanisms, response dynamics and perceptual functions of multisensory interactions in auditory cortex , 2009, Hearing Research.

[41]  J. Navarra,et al.  Assessing automaticity in audiovisual speech integration: evidence from the speeded classification task , 2004, Cognition.

[42]  Aaron R. Nidiffer,et al.  Identifying and Quantifying Multisensory Integration: A Tutorial Review , 2014, Brain Topography.

[43]  Murray MM., Cappe, C., Romei, V., Martuzzi, R. Thut, G. (In Press). Auditory-visual multisensory interactions in humans: a synthesis of findings from behavior, ERPs, fMRI, and TMS. In The New Handbook of Multisensory Processes. Edited by Barry E. Stein. MIT Press. , 2012 .

[44]  Aaron R. Nidiffer,et al.  Spatial and Temporal Features of Multisensory Processes , 2012 .

[45]  Lucas Spierer,et al.  Contributions of pitch and bandwidth to sound-induced enhancement of visual cortex excitability in humans , 2013, Cortex.

[46]  Jeff Miller,et al.  Locus of the redundant-signals effect in bimodal divided attention: A neurophysiological analysis , 2001, Perception & psychophysics.

[47]  Kunkel Jm,et al.  Spontaneous subclavain vein thrombosis: a successful combined approach of local thrombolytic therapy followed by first rib resection. , 1989 .

[48]  J. Vroomen,et al.  Electrophysiological correlates of predictive coding of auditory location in the perception of natural audiovisual events , 2012, Front. Integr. Neurosci..

[49]  S. Celebrini,et al.  Visuo-auditory interactions in the primary visual cortex of the behaving monkey: Electrophysiological evidence , 2008, BMC Neuroscience.

[50]  J. Pernier,et al.  Dynamics of cortico-subcortical cross-modal operations involved in audio-visual object detection in humans. , 2002, Cerebral cortex.

[51]  Eren Gunseli,et al.  Is a search template an ordinary working memory? Comparing electrophysiological markers of working memory maintenance for visual search and recognition , 2014, Neuropsychologia.

[52]  B. Stein,et al.  Multisensory Integration Produces an Initial Response Enhancement , 2007, Frontiers in integrative neuroscience.

[53]  Gregor Thut,et al.  Auditory–Visual Multisensory Interactions in Humans: Timing, Topography, Directionality, and Sources , 2010, The Journal of Neuroscience.

[54]  Joan López-Moliner,et al.  Sound-driven enhancement of vision: disentangling detection-level from decision-level contributions. , 2013, Journal of neurophysiology.

[55]  Joost X. Maier,et al.  Multisensory Integration of Dynamic Faces and Voices in Rhesus Monkey Auditory Cortex , 2005 .

[56]  Eveline Geiser,et al.  The role of auditory cortices in the retrieval of single‐trial auditory–visual object memories , 2015, The European journal of neuroscience.

[57]  Benjamin A. Rowland,et al.  A model of the temporal dynamics of multisensory enhancement , 2014, Neuroscience & Biobehavioral Reviews.

[58]  Pawel J. Matusz,et al.  Multi-modal distraction: Insights from children’s limited attention , 2015, Cognition.

[59]  Jan Theeuwes,et al.  Pip and pop: nonspatial auditory signals improve spatial visual search. , 2008, Journal of experimental psychology. Human perception and performance.

[60]  Micah M. Murray,et al.  Electrical neuroimaging of memory discrimination based on single-trial multisensory learning , 2012, NeuroImage.

[61]  C. Miniussi,et al.  The Functional Importance of Rhythmic Activity in the Brain , 2012, Current Biology.

[62]  Martin Eimer,et al.  Top-down control of audiovisual search by bimodal search templates. , 2013, Psychophysiology.

[63]  Tobias S. Andersen,et al.  Visual attention modulates audiovisual speech perception , 2004 .

[64]  A. King,et al.  Multisensory integration. , 1993, Science.