Mouse frontal cortex mediates additive multisensory decisions

To interpret the world and make accurate perceptual decisions, the brain must combine information across sensory modalities, For instance, it must combine vision and hearing to localize objects based on their image and sound. Probability theory suggests that evidence from multiple independent cues should be combined additively 1, but it is unclear whether mice and other mammals do this 2–6, and the cortical substrates of multisensory integration are uncertain 7. Here we show that to localize a stimulus mice combine auditory and visual spatial cues additively, a computation supported by unisensory processing in auditory and visual cortex and additive multisensory integration in frontal cortex. We developed an audiovisual localization task where mice turn a wheel to indicate the joint position of an image and a sound. Scanning optogenetic inactivation of dorsal cortex 8–11 showed that auditory and visual areas contribute unisensory information, whereas frontal cortex (secondary motor area, MOs) contributes multisensory information to the mouse’s decision, Neuropixels recordings of >10,000 neurons indicated that neural activity in MOs reflects an additive combination of visual and auditory signals. An accumulator model 1,12,13 applied to the sensory representations of MOs neurons reproduced behaviourally observed choices and reaction times. This suggests that MOs integrates information from multiple sensory cortices, providing a signal that is then transformed into a binary decision by a downstream accumulator.

[1]  Robert A Jacobs,et al.  Bayesian integration of visual and auditory signals for spatial localization. , 2003, Journal of the Optical Society of America. A, Optics, image science, and vision.

[2]  Kenneth D Harris,et al.  Spike sorting for large, dense electrode arrays , 2015, Nature Neuroscience.

[3]  Christopher R Fetsch,et al.  Neural correlates of reliability-based cue weighting during multisensory integration , 2011, Nature Neuroscience.

[4]  Nicholas A. Steinmetz,et al.  Distributed coding of choice, action, and engagement across the mouse brain , 2019, Nature.

[5]  David Raposo,et al.  Dynamic weighting of multisensory stimuli shapes decision-making in rats and humans. , 2013, Journal of vision.

[6]  Il Memming Park,et al.  Encoding and decoding in parietal cortex during sensorimotor decision-making , 2014, Nature Neuroscience.

[7]  S. Solomon,et al.  Integration of visual and whisker signals in rat superior colliculus , 2018, Scientific Reports.

[8]  Stefano Fusi,et al.  Why neurons mix: high dimensionality for higher cognition , 2016, Current Opinion in Neurobiology.

[9]  Shay Ohayon,et al.  Open Ephys: an open-source, plugin-based platform for multichannel electrophysiology , 2017, Journal of neural engineering.

[10]  D. Raab DIVISION OF PSYCHOLOGY: STATISTICAL FACILITATION OF SIMPLE REACTION TIMES* , 1962 .

[11]  A. King,et al.  Visual–auditory spatial processing in auditory cortical neurons , 2008, Brain Research.

[12]  Daeyeol Lee,et al.  Role of rodent secondary motor cortex in value-based action selection , 2011, Nature Neuroscience.

[13]  L. Abbott,et al.  Random Convergence of Olfactory Inputs in the Drosophila Mushroom Body , 2013, Nature.

[14]  Kenneth D. Harris,et al.  High-Yield Methods for Accurate Two-Alternative Visual Psychophysics in Head-Fixed Mice , 2016, bioRxiv.

[15]  Malte Bieler,et al.  Multisensory integration in rodent tactile but not visual thalamus , 2018, Scientific Reports.

[16]  Aidan P. Murphy,et al.  Audiovisual integration in macaque face patch neurons , 2021, Current Biology.

[17]  Bingni W. Brunton,et al.  Distinct relationships of parietal and prefrontal cortices to evidence accumulation , 2014, Nature.

[18]  Lawrence G. McDade,et al.  Behavioral Indices of Multisensory Integration: Orientation to Visual Cues is Affected by Auditory Stimuli , 1989, Journal of Cognitive Neuroscience.

[19]  M. Ernst,et al.  Humans integrate visual and haptic information in a statistically optimal fashion , 2002, Nature.

[20]  Christopher R Fetsch,et al.  Dynamic Reweighting of Visual and Vestibular Cues during Self-Motion Perception , 2009, The Journal of Neuroscience.

[21]  D. Burr,et al.  The Ventriloquist Effect Results from Near-Optimal Bimodal Integration , 2004, Current Biology.

[22]  A. Churchland,et al.  Lapses in perceptual decisions reflect exploration , 2021, eLife.

[23]  N. Logothetis,et al.  Visual modulation of neurons in auditory cortex. , 2008, Cerebral cortex.

[24]  Brian L Allman,et al.  Single‐unit analysis of somatosensory processing in the core auditory cortex of hearing ferrets , 2015, The European journal of neuroscience.

[25]  Petr Znamenskiy,et al.  Mesoscale cortical dynamics reflect the interaction of sensory evidence and temporal expectation during perceptual decision-making , 2019, Neuron.

[26]  Matthew T. Kaufman,et al.  A category-free neural population supports evolving demands during decision-making , 2014, Nature Neuroscience.

[27]  Chris C. Rodgers,et al.  Spike-timing-dependent ensemble encoding by non-classically responsive cortical neurons , 2019, eLife.

[28]  Shawn R. Olsen,et al.  Gain control by layer six in cortical circuits of vision , 2012, Nature.

[29]  Zengcai V. Guo,et al.  Spatiotemporal constraints on optogenetic inactivation in cortical circuits , 2019, eLife.

[30]  Alyse M. Thomas,et al.  Fully autonomous mouse behavioral and optogenetic experiments in home-cage , 2020, bioRxiv.

[31]  A Neural Circuit for Auditory Dominance over Visual Perception , 2017, Neuron.

[32]  Cyriel M. A. Pennartz,et al.  UvA-DARE ( Digital Academic Repository ) Audiovisual Integration Enhances Stimulus Detection Performance in Mice , 2018 .

[33]  Nicholas A. Steinmetz,et al.  Sensory coding and causal impact of mouse cortex in a visual decision , 2021 .

[34]  Wei Ji Ma,et al.  Bayesian inference with probabilistic population codes , 2006, Nature Neuroscience.

[35]  Christoph Kayser,et al.  Asymmetric Multisensory Interactions of Visual and Somatosensory Responses in a Region of the Rat Parietal Cortex , 2013, PloS one.

[36]  Adrian K. C. Lee,et al.  Integration of Visual Information in Auditory Cortex Promotes Auditory Scene Analysis through Multisensory Binding , 2017, Neuron.

[37]  Lindsey L. Glickfeld,et al.  Mouse Primary Visual Cortex Is Used to Detect Both Orientation and Contrast Changes , 2013, The Journal of Neuroscience.

[38]  F. Colavita Human sensory dominance , 1974 .

[39]  Nicholas A. Steinmetz,et al.  Distinct contributions of mouse cortical areas to visual discrimination , 2018 .

[40]  David Raposo,et al.  Multisensory Decision-Making in Rats and Humans , 2012, The Journal of Neuroscience.

[41]  Matthew T. Kaufman,et al.  Posterior Parietal Cortex Guides Visual Decisions in Rats , 2016, The Journal of Neuroscience.

[42]  Yong Gu,et al.  Causal Links between Dorsal Medial Superior Temporal Area Neurons and Multisensory Heading Perception , 2012, The Journal of Neuroscience.

[43]  B. Stein,et al.  Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. , 1986, Journal of neurophysiology.

[44]  Li I. Zhang,et al.  Contextual and cross-modality modulation of auditory cortical processing through pulvinar mediated suppression , 2020, eLife.

[45]  J. Gold,et al.  Banburismus and the Brain Decoding the Relationship between Sensory Stimuli, Decisions, and Reward , 2002, Neuron.

[46]  Bingni W. Brunton,et al.  Distinct effects of prefrontal and parietal cortex inactivations on an accumulation of evidence task in the rat , 2015, bioRxiv.

[47]  Jeffrey C. Erlich,et al.  A Cortical Substrate for Memory-Guided Orienting in the Rat , 2011, Neuron.

[48]  Nicholas A. Steinmetz,et al.  Striatal activity topographically reflects cortical activity , 2021, Nature.

[49]  T. Ono,et al.  Auditory thalamus integrates visual inputs into behavioral gains , 2005, Nature Neuroscience.

[50]  Ilana B. Witten,et al.  Why Seeing Is Believing: Merging Auditory and Visual Worlds , 2005, Neuron.

[51]  J. Gold,et al.  The neural basis of decision making. , 2007, Annual review of neuroscience.

[52]  J. Duhamel,et al.  Multisensory Integration in the Ventral Intraparietal Area of the Macaque Monkey , 2007, The Journal of Neuroscience.

[53]  P. Barone,et al.  Cortical and Thalamic Pathways for Multisensory and Sensorimotor Interplay , 2012 .

[54]  A. J. King,et al.  Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus , 2004, Experimental Brain Research.

[55]  Giuliano Iurilli,et al.  Sound-Driven Synaptic Inhibition in Primary Visual Cortex , 2012, Neuron.

[56]  Johannes C. Dahmen,et al.  Subcortical circuits mediate communication between primary sensory cortical areas in mice , 2020, Nature Communications.

[57]  Jessica A. Cardin,et al.  Driving fast-spiking cells induces gamma rhythm and controls sensory responses , 2009, Nature.

[58]  Matteo Carandini,et al.  Rigbox: An Open-Source Toolbox for Probing Neurons and Behavior , 2019, bioRxiv.

[59]  Ari Rosenberg,et al.  Models and processes of multisensory cue combination , 2014, Current Opinion in Neurobiology.

[60]  Uta Noppeney,et al.  Distinct Computational Principles Govern Multisensory Integration in Primary Sensory and Association Cortices , 2016, Current Biology.

[61]  Jennifer K Bizley,et al.  Where are multisensory signals combined for perceptual decision-making? , 2016, Current Opinion in Neurobiology.

[62]  Charles D. Kopec,et al.  More than Just a “Motor”: Recent Surprises from the Frontal Cortex , 2018, The Journal of Neuroscience.

[63]  Allan R. Jones,et al.  A mesoscale connectome of the mouse brain , 2014, Nature.

[64]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[65]  Chandramouli Chandrasekaran,et al.  Computational principles and models of multisensory integration , 2017, Current Opinion in Neurobiology.

[66]  Zengcai V. Guo,et al.  A motor cortex circuit for motor planning and movement , 2015, Nature.

[67]  A. Pouget,et al.  Reference frames for representing visual and tactile locations in parietal cortex , 2005, Nature Neuroscience.

[68]  A. Kwan,et al.  Secondary Motor Cortex: Where ‘Sensory’ Meets ‘Motor’ in the Rodent Frontal Cortex , 2017, Trends in Neurosciences.

[69]  Nicholas A. Steinmetz,et al.  Spontaneous behaviors drive multidimensional, brainwide activity , 2019, Science.

[70]  G. DeAngelis,et al.  Multisensory integration: psychophysics, neurophysiology, and computation , 2009, Current Opinion in Neurobiology.

[71]  G. DeAngelis,et al.  Neural correlates of multisensory cue integration in macaque MSTd , 2008, Nature Neuroscience.

[72]  A. J. King,et al.  Behavioural benefits of multisensory processing in ferrets , 2016, The European journal of neuroscience.

[73]  T. Stanford,et al.  Multisensory integration: current issues from the perspective of the single neuron , 2008, Nature Reviews Neuroscience.

[74]  Roger Ratcliff,et al.  A Theory of Memory Retrieval. , 1978 .

[75]  Justin L Gardner,et al.  Optimality and heuristics in perceptual neuroscience , 2019, Nature Neuroscience.

[76]  David W. Franklin,et al.  Computational Mechanisms of Sensorimotor Control , 2011, Neuron.

[77]  J. Driver,et al.  Multisensory Interplay Reveals Crossmodal Influences on ‘Sensory-Specific’ Brain Regions, Neural Responses, and Judgments , 2008, Neuron.

[78]  Cyriel M A Pennartz,et al.  Audiovisual Modulation in Mouse Primary Visual Cortex Depends on Cross-Modal Stimulus Configuration and Congruency , 2017, The Journal of Neuroscience.

[79]  Sergey L. Gratiy,et al.  Fully integrated silicon probes for high-density recording of neural activity , 2017, Nature.

[80]  Xiao-Jing Wang,et al.  The importance of mixed selectivity in complex cognitive tasks , 2013, Nature.

[81]  Brian Zingg,et al.  Cross-Modality Sharpening of Visual Cortical Processing through Layer-1-Mediated Inhibition and Disinhibition , 2016, Neuron.

[82]  M. Merzenich,et al.  Role of cat primary auditory cortex for sound-localization behavior. , 1984, Journal of neurophysiology.

[83]  Jeffrey C. Erlich,et al.  Requirement of Prefrontal and Midbrain Regions for Rapid Executive Control of Behavior in the Rat , 2015, Neuron.

[84]  Christoph Kayser,et al.  Causal Inference in the Multisensory Brain , 2018, Neuron.

[85]  Zengcai V. Guo,et al.  Flow of Cortical Activity Underlying a Tactile Decision in Mice , 2014, Neuron.

[86]  Yong Gu,et al.  Multisensory Convergence of Visual and Vestibular Heading Cues in the Pursuit Area of the Frontal Eye Field. , 2016, Cerebral cortex.

[87]  Henry Markram,et al.  Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations , 2002, Neural Computation.

[88]  L. Ng,et al.  The Allen Mouse Brain Common Coordinate Framework: A 3D Reference Atlas , 2020, Cell.

[89]  Matthew Johnson,et al.  Compiling machine learning programs via high-level tracing , 2018 .

[90]  G. DeAngelis,et al.  A Normalization Model of Multisensory Integration , 2011, Nature Neuroscience.

[91]  C. Spence,et al.  Visual dominance and attention: The Colavita effect revisited , 2007, Perception & psychophysics.

[92]  Mathew E. Diamond,et al.  Supralinear and Supramodal Integration of Visual and Tactile Signals in Rats: Psychophysics and Neuronal Mechanisms , 2018, Neuron.

[93]  Konrad Paul Kording,et al.  Bayesian integration in sensorimotor learning , 2004, Nature.

[94]  A. Ghazanfar,et al.  Is neocortex essentially multisensory? , 2006, Trends in Cognitive Sciences.

[95]  Elina A K Jacobs,et al.  Aberrant Cortical Activity in Multiple GCaMP6-Expressing Transgenic Mouse Lines , 2017, eNeuro.

[96]  Jennifer K. Bizley,et al.  Visual influences on ferret auditory cortex , 2009, Hearing Research.

[97]  Athanasios Alexopoulos,et al.  Mechanisms. , 2021, Deutsches Arzteblatt international.

[98]  Kenneth D. Harris,et al.  Neuropixels 2.0: A miniaturized high-density probe for stable, long-term brain recordings , 2020, Science.

[99]  M. Piché,et al.  Age-related audiovisual interactions in the superior colliculus of the rat , 2016, Neuroscience.

[100]  T. Powell,et al.  An anatomical study of converging sensory pathways within the cerebral cortex of the monkey. , 1970, Brain : a journal of neurology.

[101]  Matthew T. Kaufman,et al.  Single-trial neural dynamics are dominated by richly varied movements , 2019, Nature Neuroscience.