Computational principles and models of multisensory integration

Combining information from multiple senses creates robust percepts, speeds up responses, enhances learning, and improves detection, discrimination, and recognition. In this review, I discuss computational models and principles that provide insight into how this process of multisensory integration occurs at the behavioral and neural level. My initial focus is on drift-diffusion and Bayesian models that can predict behavior in multisensory contexts. I then highlight how recent neurophysiological and perturbation experiments provide evidence for a distributed redundant network for multisensory integration. I also emphasize studies which show that task-relevant variables in multisensory contexts are distributed in heterogeneous neural populations. Finally, I describe dimensionality reduction methods and recurrent neural network models that may help decipher heterogeneous neural populations involved in multisensory integration.

[1]  H. Bülthoff,et al.  Merging the senses into a robust percept , 2004, Trends in Cognitive Sciences.

[2]  A. Diederich Intersensory facilitation of reaction time: evaluation of counter and diffusion coactivation models , 1995 .

[3]  G. DeAngelis,et al.  Multisensory integration: psychophysics, neurophysiology, and computation , 2009, Current Opinion in Neurobiology.

[4]  Asif A Ghazanfar,et al.  Dynamic faces speed up the onset of auditory cortical spiking responses during vocal detection , 2013, Proceedings of the National Academy of Sciences.

[5]  Thomas U. Otto,et al.  Noise and Correlations in Parallel Perceptual Decision Making , 2011, Current Biology.

[6]  Byron M. Yu,et al.  Dimensionality reduction for large-scale neural recordings , 2014, Nature Neuroscience.

[7]  Christoph Kayser,et al.  Multisensory Causal Inference in the Brain , 2015, PLoS biology.

[8]  A. Diederich,et al.  Bimodal and trimodal multisensory enhancement: Effects of stimulus onset and intensity on reaction time , 2004, Perception & psychophysics.

[9]  Gregory C. DeAngelis,et al.  Bridging the gap between theories of sensory cue integration and the physiology of multisensory neurons , 2013, Nature Reviews Neuroscience.

[10]  Asif A. Ghazanfar,et al.  The Natural Statistics of Audiovisual Speech , 2009, PLoS Comput. Biol..

[11]  T. Stanford,et al.  Evaluating the Operations Underlying Multisensory Integration in the Cat Superior Colliculus , 2005, The Journal of Neuroscience.

[12]  Jonathan W. Pillow,et al.  Dissociated functional significance of decision-related activity in the primate dorsal stream , 2016, Nature.

[13]  Frédéric Crevecoeur,et al.  A perspective on multisensory integration and rapid perturbation responses , 2015, Vision Research.

[14]  David Raposo,et al.  Multisensory Decision-Making in Rats and Humans , 2012, The Journal of Neuroscience.

[15]  Konrad Paul Kording,et al.  Causal Inference in Multisensory Perception , 2007, PloS one.

[16]  W. Newsome,et al.  Context-dependent computation by recurrent dynamics in prefrontal cortex , 2013, Nature.

[17]  Jennifer K Bizley,et al.  Where are multisensory signals combined for perceptual decision-making? , 2016, Current Opinion in Neurobiology.

[18]  Wei Ji Ma,et al.  Bayesian inference with probabilistic population codes , 2006, Nature Neuroscience.

[19]  E. M. Rouiller,et al.  Multisensory anatomical pathways , 2009, Hearing Research.

[20]  C. Spence Multisensory Flavor Perception , 2015, Cell.

[21]  I. Nelken,et al.  Physiological and Anatomical Evidence for Multisensory Interactions in Auditory Cortex , 2006, Cerebral cortex.

[22]  Joost X. Maier,et al.  Multisensory Integration of Dynamic Faces and Voices in Rhesus Monkey Auditory Cortex , 2005 .

[23]  D. Knill,et al.  The Bayesian brain: the role of uncertainty in neural coding and computation , 2004, Trends in Neurosciences.

[24]  P. Mamassian,et al.  Multisensory processing in review: from physiology to behaviour. , 2010, Seeing and perceiving.

[25]  M. Shadlen,et al.  Response of Neurons in the Lateral Intraparietal Area during a Combined Visual Discrimination Reaction Time Task , 2002, The Journal of Neuroscience.

[26]  Christopher J. Cueva,et al.  Dynamics of Neural Population Responses in Prefrontal Cortex Indicate Changes of Mind on Single Trials , 2014, Current Biology.

[27]  Christian K. Machens,et al.  Behavioral / Systems / Cognitive Functional , But Not Anatomical , Separation of “ What ” and “ When ” in Prefrontal Cortex , 2009 .

[28]  Aaron R. Seitz,et al.  Benefits of multisensory learning , 2008, Trends in Cognitive Sciences.

[29]  Yong Gu,et al.  Multisensory Convergence of Visual and Vestibular Heading Cues in the Pursuit Area of the Frontal Eye Field. , 2016, Cerebral cortex.

[30]  Christopher R Fetsch,et al.  Neural correlates of reliability-based cue weighting during multisensory integration , 2011, Nature Neuroscience.

[31]  Jeff Miller,et al.  Statistical facilitation and the redundant signals effect: What are race and coactivation models? , 2016, Attention, perception & psychophysics.

[32]  W. Schwarz,et al.  Diffusion, superposition, and the redundant-targets effect , 1994 .

[33]  Jeff Miller,et al.  Divided attention: Evidence for coactivation with redundant signals , 1982, Cognitive Psychology.

[34]  M. Murray,et al.  Multisensory Integration: Flexible Use of General Operations , 2014, Neuron.

[35]  Mark W Greenlee,et al.  Multisensory processing of redundant information in go/no-go and choice responses , 2014, Attention, Perception, & Psychophysics.

[36]  Micah M. Murray,et al.  Multisensory Facilitation of Behavior in Monkeys: Effects of Stimulus Intensity , 2010, Journal of Cognitive Neuroscience.

[37]  Gerhard von der Emde,et al.  Cross-modal object recognition and dynamic weighting of sensory inputs in a fish , 2016, Proceedings of the National Academy of Sciences.

[38]  Erin L. Rich,et al.  Decoding subjective decisions from orbitofrontal cortex , 2016, Nature Neuroscience.

[39]  G. DeAngelis,et al.  Neural correlates of multisensory cue integration in macaque MSTd , 2008, Nature Neuroscience.

[40]  D. Burr,et al.  The Ventriloquist Effect Results from Near-Optimal Bimodal Integration , 2004, Current Biology.

[41]  Thomas U. Otto,et al.  Principles of Multisensory Behavior , 2013, The Journal of Neuroscience.

[42]  G. DeAngelis,et al.  How Can Single Sensory Neurons Predict Behavior? , 2015, Neuron.

[43]  Luc H. Arnal,et al.  Cortical oscillations and sensory predictions , 2012, Trends in Cognitive Sciences.

[44]  Guangyu R. Yang,et al.  Training Excitatory-Inhibitory Recurrent Neural Networks for Cognitive Tasks: A Simple and Flexible Framework , 2016, PLoS Comput. Biol..

[45]  Matthew T. Kaufman,et al.  A neural network that finds a naturalistic solution for the production of muscle activity , 2015, Nature Neuroscience.

[46]  A. Pouget,et al.  Probabilistic brains: knowns and unknowns , 2013, Nature Neuroscience.

[47]  M. Ernst,et al.  Humans integrate visual and haptic information in a statistically optimal fashion , 2002, Nature.

[48]  Aaron R. Nidiffer,et al.  Identifying and Quantifying Multisensory Integration: A Tutorial Review , 2014, Brain Topography.

[49]  Ari Rosenberg,et al.  Models and processes of multisensory cue combination , 2014, Current Opinion in Neurobiology.

[50]  U. Noppeney,et al.  Cortical Hierarchies Perform Bayesian Causal Inference in Multisensory Perception , 2015, PLoS biology.

[51]  Christoph Kayser,et al.  Spatial Organization of Multisensory Responses in Temporal Association Cortex , 2009, The Journal of Neuroscience.

[52]  Mark M Churchland,et al.  Vacillation, indecision and hesitation in moment-by-moment decoding of monkey motor cortex , 2015, eLife.

[53]  D. Raab DIVISION OF PSYCHOLOGY: STATISTICAL FACILITATION OF SIMPLE REACTION TIMES* , 1962 .

[54]  Joseph G. Makin,et al.  Learning Multisensory Integration and Coordinate Transformation via Density Estimation , 2013, PLoS Comput. Biol..

[55]  Wei Ji Ma,et al.  Neural coding of uncertainty and probability. , 2014, Annual review of neuroscience.

[56]  Yong Gu,et al.  Evidence for a Causal Contribution of Macaque Vestibular, But Not Intraparietal, Cortex to Heading Perception , 2016, The Journal of Neuroscience.

[57]  Brian E. Russ,et al.  Prefrontal Neurons Predict Choices during an Auditory Same-Different Task , 2008, Current Biology.

[58]  Katsumi Minakata,et al.  A tutorial on testing the race model inequality , 2015, Attention, Perception, & Psychophysics.

[59]  A. Ghazanfar,et al.  Is neocortex essentially multisensory? , 2006, Trends in Cognitive Sciences.

[60]  David Sussillo,et al.  Neural circuits as computational dynamical systems , 2014, Current Opinion in Neurobiology.

[61]  Lizabeth M Romanski,et al.  Representation and integration of auditory and visual stimuli in the primate ventral lateral prefrontal cortex. , 2007, Cerebral cortex.

[62]  Joost X. Maier,et al.  A Multisensory Network for Olfactory Processing , 2015, Current Biology.

[63]  John J. Foxe,et al.  Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. , 2006, Cerebral cortex.

[64]  David J. Freedman,et al.  Choice-correlated activity fluctuations underlie learning of neuronal category representation , 2015, Nature Communications.

[65]  C. Schroeder,et al.  Neuronal Oscillations and Multisensory Interaction in Primary Auditory Cortex , 2007, Neuron.

[66]  M. Sahani,et al.  Cortical control of arm movements: a dynamical systems perspective. , 2013, Annual review of neuroscience.

[67]  Markus Siegel,et al.  Cortical information flow during flexible sensorimotor decisions , 2015, Science.

[68]  Matthew T. Kaufman,et al.  A category-free neural population supports evolving demands during decision-making , 2014, Nature Neuroscience.

[69]  Andrew S. Liu,et al.  Causal contribution of primate auditory cortex to auditory perceptual decision-making , 2015, Nature Neuroscience.

[70]  Naoshige Uchida,et al.  Demixed principal component analysis of neural population data , 2014, eLife.

[71]  Xiao-Jing Wang,et al.  The importance of mixed selectivity in complex cognitive tasks , 2013, Nature.

[72]  Bruce G Cumming,et al.  Feedforward and Feedback Sources of Choice Probability in Neural Population Responses This Review Comes from a Themed Issue on Neurobiology of Cognitive Behavior Evidence for Feed-forward Models and Optimal Linear Readout? , 2022 .

[73]  Veit Stuphorn,et al.  Sequential selection of economic good and action in medial frontal cortex of 1 macaques during value-based decisions 2 3 Running title : Sequential good and action selection during decision-making 4 5 , 2015 .

[74]  Rogelio Luna,et al.  Do Sensory Cortices Process More than One Sensory Modality during Perceptual Judgments? , 2010, Neuron.

[75]  Kristin Branson,et al.  A multilevel multimodal circuit enhances action selection in Drosophila , 2015, Nature.

[76]  N. Holmes The Principle of Inverse Effectiveness in Multisensory Integration: Some Statistical Considerations , 2009, Brain Topography.

[77]  Asif A. Ghazanfar,et al.  Monkeys and Humans Share a Common Computation for Face/Voice Integration , 2011, PLoS Comput. Biol..

[78]  Dora E Angelaki,et al.  Functional Specializations of the Ventral Intraparietal Area for Multisensory Heading Discrimination , 2013, The Journal of Neuroscience.

[79]  A. Pouget,et al.  Probabilistic population codes and the exponential family of distributions. , 2007, Progress in brain research.

[80]  N. Logothetis,et al.  Visual modulation of neurons in auditory cortex. , 2008, Cerebral cortex.

[81]  Alexandre Pouget,et al.  Optimal multisensory decision-making in a reaction-time task , 2014, eLife.

[82]  W. H. Sumby,et al.  Visual contribution to speech intelligibility in noise , 1954 .

[83]  Asif A Ghazanfar,et al.  Different neural frequency bands integrate faces and voices differently in the superior temporal sulcus. , 2009, Journal of neurophysiology.

[84]  Matthew T. Kaufman,et al.  Neural population dynamics during reaching , 2012, Nature.

[85]  E. Rouiller,et al.  Multisensory Integration in Non-Human Primates during a Sensory-Motor Task , 2013, Front. Hum. Neurosci..

[86]  Jeff Miller,et al.  Timecourse of coactivation in bimodal divided attention , 1986, Perception & psychophysics.

[87]  Wei Ji Ma,et al.  Linking neurons to behavior in multisensory perception: A computational review , 2008, Brain Research.

[88]  W. Ma Organizing probabilistic models of perception , 2012, Trends in Cognitive Sciences.

[89]  Asif A Ghazanfar,et al.  Interactions between the Superior Temporal Sulcus and Auditory Cortex Mediate Dynamic Face/Voice Integration in Rhesus Monkeys , 2008, The Journal of Neuroscience.

[90]  A. Diederich,et al.  The time window of multisensory integration: relating reaction times and judgments of temporal order. , 2015, Psychological review.

[91]  Philip N. Sabes,et al.  Sensory integration for reaching: models of optimality in the context of behavior and the underlying neural circuits. , 2011, Progress in brain research.