Bayesian Cue Integration as a Developmental Outcome of Reward Mediated Learning

Average human behavior in cue combination tasks is well predicted by Bayesian inference models. As this capability is acquired over developmental timescales, the question arises, how it is learned. Here we investigated whether reward dependent learning, that is well established at the computational, behavioral, and neuronal levels, could contribute to this development. It is shown that a model free reinforcement learning algorithm can indeed learn to do cue integration, i.e. weight uncertain cues according to their respective reliabilities and even do so if reliabilities are changing. We also consider the case of causal inference where multimodal signals can originate from one or multiple separate objects and should not always be integrated. In this case, the learner is shown to develop a behavior that is closest to Bayesian model averaging. We conclude that reward mediated learning could be a driving force for the development of cue integration and causal inference.

[1]  G. J. Thomas Experimental study of the influence of vision on sound localization. , 1941 .

[2]  D. A. Grant,et al.  Acquisition and extinction of a verbal conditioned response with differing percentages of reinforcement. , 1951, Journal of experimental psychology.

[3]  I RUBINSTEIN,et al.  Some factors in probability matching. , 1959, Journal of experimental psychology.

[4]  N. L. Johnson,et al.  Linear Statistical Inference and Its Applications , 1966 .

[5]  Calyampudi R. Rao,et al.  Linear Statistical Inference and Its Applications. , 1975 .

[6]  Z. M. Fuzessery,et al.  A representation of horizontal sound location in the inferior colliculus of the mustache bat (Pteronotus p. parnellii) , 1985, Hearing Research.

[7]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[8]  D. Sparks,et al.  Population coding of saccadic eye movements by neurons in the superior colliculus , 1988, Nature.

[9]  Francis Crick,et al.  The recent excitement about neural networks , 1989, Nature.

[10]  E I Knudsen,et al.  Visual instruction of the neural map of auditory space in the developing optic tectum. , 1991, Science.

[11]  M. Landy,et al.  A perturbation analysis of depth perception from combinations of texture and motion cues , 1993, Vision Research.

[12]  S. Sloman The empirical case for two systems of reasoning. , 1996 .

[13]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[14]  M T Wallace,et al.  Development of Multisensory Neurons and Multisensory Integration in Cat Superior Colliculus , 1997, The Journal of Neuroscience.

[15]  R. Jacobs,et al.  Optimal integration of texture and motion cues to depth , 1999, Vision Research.

[16]  R. Jacobs,et al.  Experience-dependent integration of texture and motion cues to depth , 1999, Vision Research.

[17]  Li I. Zhang,et al.  Selective Presynaptic Propagation of Long-Term Potentiation in Defined Neural Networks , 2000, The Journal of Neuroscience.

[18]  W. Schultz Multiple reward signals in the brain , 2000, Nature Reviews Neuroscience.

[19]  M. Wallace,et al.  Sensory and Multisensory Responses in the Newborn Monkey Superior Colliculus , 2001, The Journal of Neuroscience.

[20]  O. Hikosaka,et al.  Modulation of saccadic eye movements by predicted reward outcome , 2001, Experimental Brain Research.

[21]  M. Ernst,et al.  Humans integrate visual and haptic information in a statistically optimal fashion , 2002, Nature.

[22]  D. Ballard,et al.  Fast Temporal Dynamics of Visual Cue Integration , 2000, Perception.

[23]  Robert A Jacobs,et al.  Bayesian integration of visual and auditory signals for spatial localization. , 2003, Journal of the Optical Society of America. A, Optics, image science, and vision.

[24]  J. O'Doherty,et al.  Encoding Predictive Reward Value in Human Amygdala and Orbitofrontal Cortex , 2003, Science.

[25]  J. Lewald,et al.  Cross-modal perceptual integration of spatially and temporally disparate auditory and visual stimuli. , 2003, Brain research. Cognitive brain research.

[26]  Daniel Kersten,et al.  Bayesian models of object perception , 2003, Current Opinion in Neurobiology.

[27]  M. Wallace,et al.  Visual Localization Ability Influences Cross-Modal Bias , 2003, Journal of Cognitive Neuroscience.

[28]  J. Saunders,et al.  Do humans optimally integrate stereo and texture information for judgments of surface slant? , 2003, Vision Research.

[29]  D. Burr,et al.  The Ventriloquist Effect Results from Near-Optimal Bimodal Integration , 2004, Current Biology.

[30]  Konrad Paul Kording,et al.  A Neuroeconomics Approach to Inferring Utility Functions in Sensorimotor Control , 2004, PLoS biology.

[31]  D. Knill,et al.  The Bayesian brain: the role of uncertainty in neural coding and computation , 2004, Trends in Neurosciences.

[32]  Karl J. Friston,et al.  Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning , 2004, Science.

[33]  H. Bülthoff,et al.  Merging the senses into a robust percept , 2004, Trends in Cognitive Sciences.

[34]  S. Nishida,et al.  Recalibration of audiovisual simultaneity , 2004, Nature Neuroscience.

[35]  A. Yuille,et al.  Object perception as Bayesian inference. , 2004, Annual review of psychology.

[36]  K. Doya,et al.  Representation of Action-Specific Reward Values in the Striatum , 2005, Science.

[37]  David C Knill,et al.  Reaching for visual cues to depth: the brain combines depth cues differently for motor control and perception. , 2005, Journal of vision.

[38]  Pieter R. Roelfsema,et al.  Attention-Gated Reinforcement Learning of Internal Representations for Classification , 2005, Neural Computation.

[39]  P. Dayan,et al.  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control , 2005, Nature Neuroscience.

[40]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[41]  Virginie van Wassenhove,et al.  Auditory-visual fusion in speech perception in children with cochlear implants. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[42]  M. Kawato,et al.  Different neural correlates of reward expectation and reward expectation error in the putamen and caudate nucleus during stimulus-action-reward association learning. , 2006, Journal of neurophysiology.

[43]  Kae Nakamura,et al.  Basal ganglia orient eyes to reward. , 2006, Journal of neurophysiology.

[44]  Wei Ji Ma,et al.  Bayesian inference with probabilistic population codes , 2006, Nature Neuroscience.

[45]  P. Dayan,et al.  Cortical substrates for exploratory decisions in humans , 2006, Nature.

[46]  R. Dolan,et al.  Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans , 2006, Nature.

[47]  E. Vaadia,et al.  Midbrain dopamine neurons encode decisions for future action , 2006, Nature Neuroscience.

[48]  Shinsuke Shimojo,et al.  Development of multisensory spatial integration and perception in humans. , 2006, Developmental science.

[49]  Konrad Paul Kording,et al.  Causal Inference in Multisensory Perception , 2007, PloS one.

[50]  Alan A. Stocker,et al.  A Bayesian Model of Conditioned Perception , 2007, NIPS.

[51]  Tim Kiemel,et al.  Development of multisensory reweighting for posture control in children , 2007, Experimental Brain Research.

[52]  F. Rösler,et al.  Early visual deprivation impairs multisensory interactions in humans , 2007, Nature Neuroscience.

[53]  N. Daw,et al.  Reinforcement Learning Signals in the Human Striatum Distinguish Learners from Nonlearners during Reward-Based Decision Making , 2007, The Journal of Neuroscience.

[54]  Marc O Ernst,et al.  Knowledge about a Common Source Can Promote Visual — Haptic Integration , 2007, Perception.

[55]  Kazuyuki Aihara,et al.  Bayesian Inference Explains Perception of Unity and Ventriloquism Aftereffect: Identification of Common Sources of Audiovisual Stimuli , 2007, Neural Computation.

[56]  Peter Dayan,et al.  Hippocampal Contributions to Control: The Third Way , 2007, NIPS.

[57]  M. Wallace,et al.  Early experience determines how the senses will interact. , 2007, Journal of neurophysiology.

[58]  Pete R. Jones,et al.  Development of Cue Integration in Human Navigation , 2008, Current Biology.

[59]  M. Sahani,et al.  Implicit knowledge of visual uncertainty guides decisions with asymmetric outcomes. , 2008, Journal of vision.

[60]  Botond Cseke,et al.  Advances in Neural Information Processing Systems 20 (NIPS 2007) , 2008 .

[61]  Samuel M. McClure,et al.  BOLD Responses Reflecting Dopaminergic Signals in the Human Ventral Tegmental Area , 2008, Science.

[62]  R. Dolan,et al.  Subliminal Instrumental Conditioning Demonstrated in the Human Brain , 2008, Neuron.

[63]  Timothy D. Hanks,et al.  Probabilistic Population Codes for Bayesian Decision Making , 2008, Neuron.

[64]  David C. Burr,et al.  Young Children Do Not Integrate Visual and Haptic Form Information , 2008, Current Biology.

[65]  Jochen Triesch,et al.  Can reinforcement learning explain the development of causal inference in multisensory integration? , 2009, 2009 IEEE 8th International Conference on Development and Learning.

[66]  A. Barto,et al.  Effect on movement selection of an evolving sensory representation: A multiple controller model of skill acquisition , 2009, Brain Research.

[67]  P. Berkes,et al.  Statistically Optimal Perception and Learning: from Behavior to Neural Representations , 2022 .

[68]  Antonio Rangel,et al.  Neural computations associated with goal-directed choice , 2010, Current Opinion in Neurobiology.

[69]  M. Nardini,et al.  Fusion of visual cues is not mandatory in children , 2010, Proceedings of the National Academy of Sciences.

[70]  Ulrik R. Beierholm,et al.  Probability Matching as a Computational Strategy Used in Perception , 2010, PLoS Comput. Biol..

[71]  Jochen Triesch,et al.  Computational Modeling of Multisensory Object Perception , 2010 .

[72]  Shih-Chii Liu,et al.  Perceptron learning rule derived from spike-frequency adaptation and spike-time-dependent plasticity , 2010, Proceedings of the National Academy of Sciences.

[73]  G. Sandini,et al.  Poor Haptic Orientation Discrimination in Nonsighted Children May Reflect Disruption of Cross-Sensory Calibration , 2010, Current Biology.

[74]  Ulrik R. Beierholm,et al.  Causal inference in perception , 2010, Trends in Cognitive Sciences.

[75]  John J. Foxe,et al.  The development of audiovisual multisensory integration across childhood and early adolescence: a high-density electrical mapping study. , 2011, Cerebral cortex.

[76]  David R. Wozny,et al.  Recalibration of Auditory Space following Milliseconds of Cross-Modal Discrepancy , 2011, The Journal of Neuroscience.

[77]  P. Barone,et al.  Effect of Audiovisual Training on Monaural Spatial Hearing in Horizontal Plane , 2011, PloS one.

[78]  P. Dayan,et al.  Model-based influences on humans’ choices and striatal prediction errors , 2011, Neuron.

[79]  Amir Dezfouli,et al.  Speed/Accuracy Trade-Off between the Habitual and the Goal-Directed Processes , 2011, PLoS Comput. Biol..

[80]  Pawan Sinha,et al.  Corrigendum: The newly sighted fail to match seen with felt , 2011, Nature Neuroscience.