Decentralized Multisensory Information Integration in Neural Systems

How multiple sensory cues are integrated in neural circuitry remains a challenge. The common hypothesis is that information integration might be accomplished in a dedicated multisensory integration area receiving feedforward inputs from the modalities. However, recent experimental evidence suggests that it is not a single multisensory brain area, but rather many multisensory brain areas that are simultaneously involved in the integration of information. Why many mutually connected areas should be needed for information integration is puzzling. Here, we investigated theoretically how information integration could be achieved in a distributed fashion within a network of interconnected multisensory areas. Using biologically realistic neural network models, we developed a decentralized information integration system that comprises multiple interconnected integration areas. Studying an example of combining visual and vestibular cues to infer heading direction, we show that such a decentralized system is in good agreement with anatomical evidence and experimental observations. In particular, we show that this decentralized system can integrate information optimally. The decentralized system predicts that optimally integrated information should emerge locally from the dynamics of the communication between brain areas and sheds new light on the interpretation of the connectivity between multisensory brain areas. SIGNIFICANCE STATEMENT To extract information reliably from ambiguous environments, the brain integrates multiple sensory cues, which provide different aspects of information about the same entity of interest. Here, we propose a decentralized architecture for multisensory integration. In such a system, no processor is in the center of the network topology and information integration is achieved in a distributed manner through reciprocally connected local processors. Through studying the inference of heading direction with visual and vestibular cues, we show that the decentralized system can integrate information optimally, with the reciprocal connections between processers determining the extent of cue integration. Our model reproduces known multisensory integration behaviors observed in experiments and sheds new light on our understanding of how information is integrated in the brain.

[1]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[2]  Leslie G. Ungerleider,et al.  Pathways for motion analysis: Cortical connections of the medial superior temporal and fundus of the superior temporal visual areas in the macaque , 1990, The Journal of comparative neurology.

[3]  James J. Clark,et al.  Data Fusion for Sensory Information Processing Systems , 1990 .

[4]  Leslie G. Ungerleider,et al.  Organization of visual inputs to the inferior temporal and posterior parietal cortex in macaques , 1991, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[5]  J. Movshon,et al.  The analysis of visual motion: a comparison of neuronal and psychophysical performance , 1992, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[6]  H. Sompolinsky,et al.  Theory of orientation tuning in visual cortex. , 1995, Proceedings of the National Academy of Sciences of the United States of America.

[7]  K. Zhang,et al.  Representation of spatial orientation by the intrinsic dynamics of the head-direction cell ensemble: a theory , 1996, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[8]  J. Movshon,et al.  Linearity and Normalization in Simple Cells of the Macaque Primary Visual Cortex , 1997, The Journal of Neuroscience.

[9]  B L McNaughton,et al.  Path Integration and Cognitive Mapping in a Continuous Attractor Neural Network Model , 1997, The Journal of Neuroscience.

[10]  G. Edelman,et al.  Consciousness and Complexity , 1998 .

[11]  R. J. van Beers,et al.  Integration of proprioceptive and visual position-information: An experimentally supported model. , 1999, Journal of neurophysiology.

[12]  R. Jacobs,et al.  Optimal integration of texture and motion cues to depth , 1999, Vision Research.

[13]  A. Pouget,et al.  Reading population codes: a neural implementation of ideal observers , 1999, Nature Neuroscience.

[14]  P. Goldman-Rakic,et al.  Synaptic mechanisms and network dynamics underlying spatial working memory in a cortical network model. , 2000, Cerebral cortex.

[15]  Pramod K. Varshney,et al.  Multisensor Data Fusion , 1997, IEA/AIE.

[16]  D. V. van Essen,et al.  Corticocortical connections of visual, sensorimotor, and multimodal processing areas in the parietal lobe of the macaque monkey , 2000, The Journal of comparative neurology.

[17]  A. Pouget,et al.  Efficient computation and cue integration with noisy population codes , 2001, Nature Neuroscience.

[18]  M. Ernst,et al.  Humans integrate visual and haptic information in a statistically optimal fashion , 2002, Nature.

[19]  James M. Hillis,et al.  Combining Sensory Information: Mandatory Fusion Within, but Not Between, Senses , 2002, Science.

[20]  H Misslisch,et al.  Temporal properties of optic flow responses in the ventral intraparietal area , 2002, Visual Neuroscience.

[21]  Si Wu,et al.  Population Coding and Decoding in a Neural Field: A Computational Study , 2002, Neural Computation.

[22]  Alexandre Pouget,et al.  A computational perspective on the neural basis of multisensory spatial representations , 2002, Nature Reviews Neuroscience.

[23]  A. Berthoz,et al.  Visuo-vestibular interaction in the reconstruction of travelled trajectories , 2003, Experimental Brain Research.

[24]  R. Zemel,et al.  Inference and computation with population codes. , 2003, Annual review of neuroscience.

[25]  D. Burr,et al.  The Ventriloquist Effect Results from Near-Optimal Bimodal Integration , 2004, Current Biology.

[26]  Konrad Paul Körding,et al.  The loss function of sensorimotor learning. , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[27]  Alexandre Pouget,et al.  Bayesian multisensory integration and cross-modal spatial links , 2004, Journal of Physiology-Paris.

[28]  Ulrik R Beierholm,et al.  Sound-induced flash illusion as an optimal percept , 2005, Neuroreport.

[29]  A. Pouget,et al.  Reference frames for representing visual and tactile locations in parietal cortex , 2005, Nature Neuroscience.

[30]  Wei Ji Ma,et al.  Bayesian inference with probabilistic population codes , 2006, Nature Neuroscience.

[31]  Jean-Pierre Bresciani,et al.  Vision and touch are automatically integrated for the perception of sequences of events. , 2006, Journal of vision.

[32]  Neil W. Roach,et al.  Resolving multisensory conflict: a strategy for balancing the costs and benefits of audio-visual integration , 2006, Proceedings of the Royal Society B: Biological Sciences.

[33]  Marc O. Ernst,et al.  A Bayesian view on multimodal cue integration , 2006 .

[34]  Konrad Paul Kording,et al.  Causal Inference in Multisensory Perception , 2007, PloS one.

[35]  Justin L. Vincent,et al.  Intrinsic functional architecture in the anaesthetized monkey brain , 2007, Nature.

[36]  Kazuyuki Aihara,et al.  Bayesian Inference Explains Perception of Unity and Ventriloquism Aftereffect: Identification of Common Sources of Audiovisual Stimuli , 2007, Neural Computation.

[37]  David R. Wozny,et al.  Human trimodal perception follows optimal statistical inference. , 2008, Journal of vision.

[38]  T. Stanford,et al.  A neural network model of multisensory integration also accounts for unisensory integration in superior colliculus , 2008, Brain Research.

[39]  John J. Foxe,et al.  Crossmodal binding through neural coherence: implications for multisensory processing , 2008, Trends in Neurosciences.

[40]  G. DeAngelis,et al.  Multisensory Integration in Macaque Visual Cortex Depends on Cue Reliability , 2008, Neuron.

[41]  Mauro Ursino,et al.  Multisensory integration in the superior colliculus: a neural network model , 2009, Journal of Computational Neuroscience.

[42]  Romi Nijhawan,et al.  Behavioral significance of motion direction causes anisotropic flash-lag, flash-drag, flash-repulsion, and movement-mislocalization effects. , 2008, Journal of vision.

[43]  G. DeAngelis,et al.  Neural correlates of multisensory cue integration in macaque MSTd , 2008, Nature Neuroscience.

[44]  T. Stanford,et al.  Multisensory integration: current issues from the perspective of the single neuron , 2008, Nature Reviews Neuroscience.

[45]  Mauro Ursino,et al.  A theoretical study of multisensory integration in the superior colliculus by a neural network model , 2008, Neural Networks.

[46]  Si Wu,et al.  Dynamics and Computation of Continuous Attractors , 2008, Neural Computation.

[47]  Christopher R Fetsch,et al.  Dynamic Reweighting of Visual and Vestibular Cues during Self-Motion Perception , 2009, The Journal of Neuroscience.

[48]  Anil K. Seth,et al.  Consciousness and Complexity , 2022 .

[49]  Si Wu,et al.  A Moving Bump in a Continuous Manifold: A Comprehensive Study of the Tracking Dynamics of Continuous Attractor Neural Networks , 2008, Neural Computation.

[50]  Shinsuke Shimojo,et al.  Mutually contradictory percepts in motion processing , 2010 .

[51]  Gregory C. DeAngelis,et al.  A Comparison of Vestibular Spatiotemporal Tuning in Macaque Parietoinsular Vestibular Cortex, Ventral Intraparietal Area, and Medial Superior Temporal Area , 2011, The Journal of Neuroscience.

[52]  A. Pouget,et al.  Marginalization in Neural Circuits with Divisive Normalization , 2011, The Journal of Neuroscience.

[53]  Dora E Angelaki,et al.  Convergence of Vestibular and Visual Self-Motion Signals in an Area of the Posterior Sylvian Fissure , 2011, The Journal of Neuroscience.

[54]  Philip N. Sabes,et al.  Sensory integration for reaching: models of optimality in the context of behavior and the underlying neural circuits. , 2011, Progress in brain research.

[55]  G. DeAngelis,et al.  A Normalization Model of Multisensory Integration , 2011, Nature Neuroscience.

[56]  Si Wu,et al.  Neural Information Processing with Feedback Modulations , 2012, Neural Computation.

[57]  Yong Gu,et al.  Causal Links between Dorsal Medial Superior Temporal Area Neurons and Multisensory Heading Perception , 2012, The Journal of Neuroscience.

[58]  Gustavo Deco,et al.  Stimulus-dependent variability and noise correlations in cortical MT neurons , 2013, Proceedings of the National Academy of Sciences.

[59]  Gregory C. DeAngelis,et al.  Bridging the gap between theories of sensory cue integration and the physiology of multisensory neurons , 2013, Nature Reviews Neuroscience.

[60]  Zachary P. Kilpatrick,et al.  Interareal coupling reduces encoding variability in multi-area models of spatial working memory , 2013, Front. Comput. Neurosci..

[61]  Dora E Angelaki,et al.  Functional Specializations of the Ventral Intraparietal Area for Multisensory Heading Discrimination , 2013, The Journal of Neuroscience.

[62]  A. Compte,et al.  Bump attractor dynamics in prefrontal cortex explains behavioral precision in spatial working memory , 2014, Nature Neuroscience.

[63]  James L. McClelland,et al.  Parallel Distributed Processing at 25: Further Explorations in the Microstructure of Cognition , 2014, Cogn. Sci..

[64]  A. Pouget,et al.  Information-limiting correlations , 2014, Nature Neuroscience.

[65]  Yong Gu,et al.  Multisensory Convergence of Visual and Vestibular Heading Cues in the Pursuit Area of the Frontal Eye Field. , 2016, Cerebral cortex.