Cortical Hierarchies Perform Bayesian Causal Inference in Multisensory Perception

To form a veridical percept of the environment, the brain needs to integrate sensory signals from a common source but segregate those from independent sources. Thus, perception inherently relies on solving the “causal inference problem.” Behaviorally, humans solve this problem optimally as predicted by Bayesian Causal Inference; yet, the underlying neural mechanisms are unexplored. Combining psychophysics, Bayesian modeling, functional magnetic resonance imaging (fMRI), and multivariate decoding in an audiovisual spatial localization task, we demonstrate that Bayesian Causal Inference is performed by a hierarchy of multisensory processes in the human brain. At the bottom of the hierarchy, in auditory and visual areas, location is represented on the basis that the two signals are generated by independent sources (= segregation). At the next stage, in posterior intraparietal sulcus, location is estimated under the assumption that the two signals are from a common source (= forced fusion). Only at the top of the hierarchy, in anterior intraparietal sulcus, the uncertainty about the causal structure of the world is taken into account and sensory signals are combined as predicted by Bayesian Causal Inference. Characterizing the computational operations of signal interactions reveals the hierarchical nature of multisensory perception in human neocortex. It unravels how the brain accomplishes Bayesian Causal Inference, a statistical computation fundamental for perception and cognition. Our results demonstrate how the brain combines information in the face of uncertainty about the underlying causal structure of the world.

[1]  Steven A. Hillyard,et al.  Audio-visual synchrony modulates the ventriloquist illusion and its neural/spatial representation in the auditory cortex , 2014, NeuroImage.

[2]  Uta Noppeney,et al.  Temporal prediction errors in visual and auditory cortices , 2014, Current Biology.

[3]  Gregory C. DeAngelis,et al.  Bridging the gap between theories of sensory cue integration and the physiology of multisensory neurons , 2013, Nature Reviews Neuroscience.

[4]  Christopher R Fetsch,et al.  Neural correlates of reliability-based cue weighting during multisensory integration , 2011, Nature Neuroscience.

[5]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[6]  U. Noppeney,et al.  Audiovisual Synchrony Improves Motion Discrimination via Enhanced Connectivity between Early Visual and Auditory Areas , 2010, The Journal of Neuroscience.

[7]  D. Senkowski,et al.  The multifaceted interplay between attention and multisensory integration , 2010, Trends in Cognitive Sciences.

[8]  Ulrik R. Beierholm,et al.  Causal inference in perception , 2010, Trends in Cognitive Sciences.

[9]  Ulrik R. Beierholm,et al.  Probability Matching as a Computational Strategy Used in Perception , 2010, PLoS Comput. Biol..

[10]  U. Noppeney,et al.  Distinct Functional Contributions of Primary Sensory and Association Areas to Audiovisual Integration in Object Categorization , 2010, The Journal of Neuroscience.

[11]  S. Kastner,et al.  Mechanisms of Spatial Attention Control in Frontal and Parietal Cortex , 2010, The Journal of Neuroscience.

[12]  Karl J. Friston,et al.  Bayesian model selection for group studies , 2009, NeuroImage.

[13]  G. DeAngelis,et al.  Multisensory Integration in Macaque Visual Cortex Depends on Cue Reliability , 2008, Neuron.

[14]  J. Rieger,et al.  Audiovisual Temporal Correspondence Modulates Human Multisensory Superior Temporal Sulcus Plus Primary Sensory Cortices , 2007, The Journal of Neuroscience.

[15]  S. Hillyard,et al.  Neural Basis of the Ventriloquist Illusion , 2007, Current Biology.

[16]  Konrad Paul Kording,et al.  Causal Inference in Multisensory Perception , 2007, PloS one.

[17]  J. Kaiser,et al.  Object Familiarity and Semantic Congruency Modulate Responses in Cortical Audiovisual Integration Areas , 2007, The Journal of Neuroscience.

[18]  Lotfi B Merabet,et al.  Visual Topography of Human Intraparietal Sulcus , 2007, The Journal of Neuroscience.

[19]  Christoph Kayser,et al.  Functional Imaging Reveals Visual Modulation of Specific Fields in Auditory Cortex , 2007, The Journal of Neuroscience.

[20]  C. Schroeder,et al.  Neuronal Oscillations and Multisensory Interaction in Primary Auditory Cortex , 2007, Neuron.

[21]  M. Kenward,et al.  An Introduction to the Bootstrap , 2007 .

[22]  Wei Ji Ma,et al.  Bayesian inference with probabilistic population codes , 2006, Nature Neuroscience.

[23]  Neil W. Roach,et al.  Resolving multisensory conflict: a strategy for balancing the costs and benefits of audio-visual integration , 2006, Proceedings of the Royal Society B: Biological Sciences.

[24]  A. Ghazanfar,et al.  Is neocortex essentially multisensory? , 2006, Trends in Cognitive Sciences.

[25]  Simon B. Eickhoff,et al.  A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data , 2005, NeuroImage.

[26]  James M. Hillis,et al.  Slant from texture and disparity cues: optimal cue combination. , 2004, Journal of vision.

[27]  S. Yantis,et al.  Control of Attention Shifts between Vision and Audition in Human Cortex , 2004, The Journal of Neuroscience.

[28]  B. Argall,et al.  Unraveling multisensory integration: patchy organization within human STS multisensory cortex , 2004, Nature Neuroscience.

[29]  M. Wallace,et al.  Unifying multisensory signals across time and space , 2004, Experimental Brain Research.

[30]  D. Burr,et al.  The Ventriloquist Effect Results from Near-Optimal Bimodal Integration , 2004, Current Biology.

[31]  J. Saunders,et al.  Do humans optimally integrate stereo and texture information for judgments of surface slant? , 2003, Vision Research.

[32]  Emiliano Macaluso,et al.  Multimodal Spatial Representations Engaged in Human Parietal Cortex during Both Saccadic and Manual Spatial Orienting , 2003, Current Biology.

[33]  D. Wolpert,et al.  When Feeling Is More Important Than Seeing in Sensorimotor Adaptation , 2002, Current Biology.

[34]  M. Ernst,et al.  Humans integrate visual and haptic information in a statistically optimal fashion , 2002, Nature.

[35]  C. Avendano,et al.  The CIPIC HRTF database , 2001, Proceedings of the 2001 IEEE Workshop on the Applications of Signal Processing to Audio and Acoustics (Cat. No.01TH8575).

[36]  J. Rauschecker,et al.  Functional Specialization in Rhesus Monkey Auditory Cortex , 2001, Science.

[37]  John J. Foxe,et al.  Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping. , 2000, Brain research. Cognitive brain research.

[38]  R. Jacobs,et al.  Optimal integration of texture and motion cues to depth , 1999, Vision Research.

[39]  M. Goldberg,et al.  Space and attention in parietal cortex. , 1999, Annual review of neuroscience.

[40]  Anders M. Dale,et al.  Cortical Surface-Based Analysis I. Segmentation and Surface Reconstruction , 1999, NeuroImage.

[41]  M. Goldberg,et al.  The representation of visual salience in monkey parietal cortex , 1998, Nature.

[42]  D H Brainard,et al.  The Psychophysics Toolbox. , 1997, Spatial vision.

[43]  J W Belliveau,et al.  Borders of multiple visual areas in humans revealed by functional magnetic resonance imaging. , 1995, Science.

[44]  M. Landy,et al.  Measurement and modeling of depth cue combination: in defense of weak fusion , 1995, Vision Research.

[45]  A. Raftery Bayesian Model Selection in Social Research , 1995 .

[46]  Karl J. Friston,et al.  Statistical parametric maps in functional imaging: A general linear approach , 1994 .

[47]  N. Nagelkerke,et al.  A note on a general definition of the coefficient of determination , 1991 .

[48]  Leslie G. Ungerleider,et al.  Object vision and spatial vision: two cortical pathways , 1983, Trends in Neurosciences.

[49]  P. Bertelson,et al.  Cross-modal bias and perceptual fusion with auditory-visual spatial discordance , 1981, Perception & psychophysics.