Sensory reliability shapes perceptual inference via two mechanisms.

To obtain a coherent percept of the environment, the brain should integrate sensory signals from common sources and segregate those from independent sources. Recent research has demonstrated that humans integrate audiovisual information during spatial localization consistent with Bayesian Causal Inference (CI). However, the decision strategies that human observers employ for implicit and explicit CI remain unclear. Further, despite the key role of sensory reliability in multisensory integration, Bayesian CI has never been evaluated across a wide range of sensory reliabilities. This psychophysics study presented participants with spatially congruent and discrepant audiovisual signals at four levels of visual reliability. Participants localized the auditory signals (implicit CI) and judged whether auditory and visual signals came from common or independent sources (explicit CI). Our results demonstrate that humans employ model averaging as a decision strategy for implicit CI; they report an auditory spatial estimate that averages the spatial estimates under the two causal structures weighted by their posterior probabilities. Likewise, they explicitly infer a common source during the common-source judgment when the posterior probability for a common source exceeds a fixed threshold of 0.5. Critically, sensory reliability shapes multisensory integration in Bayesian CI via two distinct mechanisms: First, higher sensory reliability sensitizes humans to spatial disparity and thereby sharpens their multisensory integration window. Second, sensory reliability determines the relative signal weights in multisensory integration under the assumption of a common source. In conclusion, our results demonstrate that Bayesian CI is fundamental for integrating signals of variable reliabilities.

[1]  Ulrik R. Beierholm,et al.  Probability Matching as a Computational Strategy Used in Perception , 2010, PLoS Comput. Biol..

[2]  Robert A Jacobs,et al.  Bayesian integration of visual and auditory signals for spatial localization. , 2003, Journal of the Optical Society of America. A, Optics, image science, and vision.

[3]  A. Raftery Bayesian Model Selection in Social Research , 1995 .

[4]  U. Noppeney,et al.  Cortical Hierarchies Perform Bayesian Causal Inference in Multisensory Perception , 2015, PLoS biology.

[5]  Christopher R Fetsch,et al.  Neural correlates of reliability-based cue weighting during multisensory integration , 2011, Nature Neuroscience.

[6]  Gregory C. DeAngelis,et al.  Bridging the gap between theories of sensory cue integration and the physiology of multisensory neurons , 2013, Nature Reviews Neuroscience.

[7]  A. Yuille,et al.  Bayesian decision theory and psychophysics , 1996 .

[8]  Marc O. Ernst,et al.  A Bayesian view on multimodal cue integration , 2006 .

[9]  F L Wightman,et al.  Headphone simulation of free-field listening. II: Psychophysical validation. , 1989, The Journal of the Acoustical Society of America.

[10]  Denis G. Pelli,et al.  ECVP '07 Abstracts , 2007, Perception.

[11]  K. Stanovich,et al.  Heuristics and Biases: Individual Differences in Reasoning: Implications for the Rationality Debate? , 2002 .

[12]  Kazuyuki Aihara,et al.  Bayesian Inference Explains Perception of Unity and Ventriloquism Aftereffect: Identification of Common Sources of Audiovisual Stimuli , 2007, Neural Computation.

[13]  U. Noppeney,et al.  Audiovisual Synchrony Improves Motion Discrimination via Enhanced Connectivity between Early Visual and Auditory Areas , 2010, The Journal of Neuroscience.

[14]  Emiliano Ricciardi,et al.  The neural mechanisms of reliability weighted integration of shape information from vision and touch , 2012, NeuroImage.

[15]  D. H. Warren,et al.  Immediate perceptual response to intersensory discrepancy. , 1980, Psychological bulletin.

[16]  Ahna R Girshick,et al.  Probabilistic combination of slant information: weighted averaging and robustness as optimal percepts. , 2009, Journal of vision.

[17]  D. Burr,et al.  The Ventriloquist Effect Results from Near-Optimal Bimodal Integration , 2004, Current Biology.

[18]  Jean-Pierre Bresciani,et al.  Vision and touch are automatically integrated for the perception of sequences of events. , 2006, Journal of vision.

[19]  Ladan Shams,et al.  Bayesian priors are encoded independently from likelihoods in human multisensory perception. , 2009, Journal of vision.

[20]  N. Nagelkerke,et al.  A note on a general definition of the coefficient of determination , 1991 .

[21]  Karl J. Friston,et al.  Statistical parametric maps in functional imaging: A general linear approach , 1994 .

[22]  Konrad Paul Kording,et al.  Causal Inference in Multisensory Perception , 2007, PloS one.

[23]  M. Ernst,et al.  Humans integrate visual and haptic information in a statistically optimal fashion , 2002, Nature.

[24]  F L Wightman,et al.  Localization using nonindividualized head-related transfer functions. , 1993, The Journal of the Acoustical Society of America.

[25]  Alan A. Stocker,et al.  A Bayesian Model of Conditioned Perception , 2007, NIPS.

[26]  C. Avendano,et al.  The CIPIC HRTF database , 2001, Proceedings of the 2001 IEEE Workshop on the Applications of Signal Processing to Audio and Acoustics (Cat. No.01TH8575).

[27]  M. Landy,et al.  Weighted linear cue combination with possibly correlated error , 2003, Vision Research.

[28]  Wei Ji Ma,et al.  Bayesian inference with probabilistic population codes , 2006, Nature Neuroscience.

[29]  M. Wallace,et al.  Unifying multisensory signals across time and space , 2004, Experimental Brain Research.

[30]  J. Lewald,et al.  Cross-modal perceptual integration of spatially and temporally disparate auditory and visual stimuli. , 2003, Brain research. Cognitive brain research.

[31]  D. Knill Robust cue integration: a Bayesian model and evidence from cue-conflict studies with stereoscopic and figure cues to slant. , 2007, Journal of vision.

[32]  R. Jacobs,et al.  Optimal integration of texture and motion cues to depth , 1999, Vision Research.

[33]  G. DeAngelis,et al.  Multisensory Integration in Macaque Visual Cortex Depends on Cue Reliability , 2008, Neuron.

[34]  Karl J. Friston,et al.  Comparing Families of Dynamic Causal Models , 2010, PLoS Comput. Biol..

[35]  Richard S. Zemel,et al.  Characterizing response behavior in multisensory perception with conflicting cues , 2008, NIPS.

[36]  Uta Noppeney,et al.  Temporal prediction errors in visual and auditory cortices , 2014, Current Biology.

[37]  S. Gepshtein,et al.  The combination of vision and touch depends on spatial proximity. , 2005, Journal of vision.

[38]  M. Beauchamp,et al.  Neural Substrates of Reliability-Weighted Visual-Tactile Multisensory Integration , 2010, Front. Syst. Neurosci..

[39]  Neil W. Roach,et al.  Resolving multisensory conflict: a strategy for balancing the costs and benefits of audio-visual integration , 2006, Proceedings of the Royal Society B: Biological Sciences.

[40]  W. Gaissmaier,et al.  The smart potential behind probability matching , 2008, Cognition.

[41]  D H Brainard,et al.  The Psychophysics Toolbox. , 1997, Spatial vision.

[42]  G. Recanzone,et al.  Temporal and spatial dependency of the ventriloquism effect , 2001, Neuroreport.

[43]  Joshua B. Tenenbaum,et al.  Causal inference in sensorimotor integration , 2006, NIPS.

[44]  J. Saunders,et al.  Do humans optimally integrate stereo and texture information for judgments of surface slant? , 2003, Vision Research.

[45]  J. Movshon,et al.  A new perceptual illusion reveals mechanisms of sensory decoding , 2007, Nature.

[46]  Ulrik R. Beierholm,et al.  Causal inference in perception , 2010, Trends in Cognitive Sciences.

[47]  U. Noppeney,et al.  Perceptual Decisions Formed by Accumulation of Audiovisual Evidence in Prefrontal Cortex , 2010, The Journal of Neuroscience.

[48]  A. Tolias,et al.  Trial-to-trial, uncertainty-based adjustment of decision boundaries in visual categorization , 2013, Proceedings of the National Academy of Sciences.