How instructed knowledge modulates the neural systems of reward learning

Recent research in neuroeconomics has demonstrated that the reinforcement learning model of reward learning captures the patterns of both behavioral performance and neural responses during a range of economic decision-making tasks. However, this powerful theoretical model has its limits. Trial-and-error is only one of the means by which individuals can learn the value associated with different decision options. Humans have also developed efficient, symbolic means of communication for learning without the necessity for committing multiple errors across trials. In the present study, we observed that instructed knowledge of cue-reward probabilities improves behavioral performance and diminishes reinforcement learning-related blood-oxygen level-dependent (BOLD) responses to feedback in the nucleus accumbens, ventromedial prefrontal cortex, and hippocampal complex. The decrease in BOLD responses in these brain regions to reward-feedback signals was functionally correlated with activation of the dorsolateral prefrontal cortex (DLPFC). These results suggest that when learning action values, participants use the DLPFC to dynamically adjust outcome responses in valuation regions depending on the usefulness of action-outcome information.

[1]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[2]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[3]  B. Richmond,et al.  Learning motivational significance of visual cues for reward schedules requires rhinal cortex , 2000, Nature Neuroscience.

[4]  B. Richmond,et al.  Response differences in monkey TE and perirhinal cortex: stimulus association related to reward schedules. , 2000, Journal of neurophysiology.

[5]  L. Nystrom,et al.  Tracking the hemodynamic responses to reward and punishment in the striatum. , 2000, Journal of neurophysiology.

[6]  D. Ariely,et al.  Beautiful Faces Have Variable Reward Value fMRI and Behavioral Evidence , 2001, Neuron.

[7]  E. Miller,et al.  An integrative theory of prefrontal cortex function. , 2001, Annual review of neuroscience.

[8]  D. Kahneman,et al.  Functional Imaging of Neural Responses to Expectancy and Experience of Monetary Gains and Losses tasks with monetary payoffs , 2001 .

[9]  E. Rolls,et al.  Abstract reward and punishment representations in the human orbitofrontal cortex , 2001, Nature Neuroscience.

[10]  Brian Knutson,et al.  Dissociation of reward anticipation and outcome with event-related fMRI , 2001, Neuroreport.

[11]  E. Rolls,et al.  Representation of pleasant and aversive taste in the human brain. , 2001, Journal of neurophysiology.

[12]  G. Glover,et al.  Dissociated neural representations of intensity and valence in human olfaction , 2003, Nature Neuroscience.

[13]  M. Mesulam,et al.  Dissociation of Neural Representation of Intensity and Affective Valuation in Human Gustation , 2003, Neuron.

[14]  Samuel M. McClure,et al.  A computational substrate for incentive salience , 2003, Trends in Neurosciences.

[15]  Samuel M. McClure,et al.  Temporal Prediction Errors in a Passive Learning Task Activate Human Striatum , 2003, Neuron.

[16]  Karl J. Friston,et al.  Temporal Difference Models and Reward-Related Learning in the Human Brain , 2003, Neuron.

[17]  Jonathan D. Cohen,et al.  The Neural Basis of Economic Decision-Making in the Ultimatum Game , 2003, Science.

[18]  Samuel M. McClure,et al.  Neural Correlates of Behavioral Preference for Culturally Familiar Drinks , 2004, Neuron.

[19]  Karl J. Friston,et al.  Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning , 2004, Science.

[20]  D. D. de Quervain,et al.  The Neural Basis of Altruistic Punishment , 2004, Science.

[21]  Samuel M. McClure,et al.  Separate Neural Systems Value Immediate and Delayed Monetary Rewards , 2004, Science.

[22]  G. Pagnoni,et al.  Human Striatal Responses to Monetary Reward Depend On Saliency , 2004, Neuron.

[23]  S. Quartz,et al.  Getting to Know You: Reputation and Trust in a Two-Person Economic Exchange , 2005, Science.

[24]  Camelia M. Kuhnen,et al.  The Neural Basis of Financial Risk Taking , 2005, Neuron.

[25]  P. Dayan,et al.  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control , 2005, Nature Neuroscience.

[26]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[27]  J. Gross,et al.  The cognitive control of emotion , 2005, Trends in Cognitive Sciences.

[28]  S. Inati,et al.  An fMRI study of reward-related probability learning , 2005, NeuroImage.

[29]  C. Padoa-Schioppa,et al.  Neurons in the orbitofrontal cortex encode economic value , 2006, Nature.

[30]  Jonathan D. Cohen,et al.  Imaging valuation models in human choice. , 2006, Annual review of neuroscience.

[31]  Keiji Tanaka,et al.  Reward Association Affects Neuronal Responses to Visual Stimuli in Macaque TE and Perirhinal Cortices , 2006, The Journal of Neuroscience.

[32]  Kenji Doya,et al.  Brain mechanism of reward prediction under predictable and unpredictable environmental dynamics , 2006, Neural Networks.

[33]  Samuel M. McClure,et al.  Policy Adjustment in a Dynamic Economic Game , 2006, PloS one.

[34]  R. Cabeza,et al.  Cognitive neuroscience of emotional memory , 2006, Nature Reviews Neuroscience.

[35]  E. Fehr,et al.  Resisting the Power of Temptations , 2007, Annals of the New York Academy of Sciences.

[36]  N. Daw,et al.  Reinforcement Learning Signals in the Human Striatum Distinguish Learners from Nonlearners during Reward-Based Decision Making , 2007, The Journal of Neuroscience.

[37]  M. Delgado,et al.  Reward‐Related Responses in the Human Striatum , 2007, Annals of the New York Academy of Sciences.

[38]  Samuel M. McClure,et al.  Short-term memory traces for action bias in human reinforcement learning , 2007, Brain Research.

[39]  U. Fischbacher,et al.  The Neural Signature of Social Norm Compliance , 2007, Neuron.

[40]  K. Sakai Task set and prefrontal cortex. , 2008, Annual review of neuroscience.

[41]  Colin Camerer,et al.  A framework for studying the neurobiology of value-based decision making , 2008, Nature Reviews Neuroscience.

[42]  Peter Bossaerts,et al.  Neural correlates of mentalizing-related computations during strategic interactions in humans , 2008, Proceedings of the National Academy of Sciences.

[43]  Colin Camerer,et al.  Dissociating the Role of the Orbitofrontal Cortex and the Striatum in the Computation of Goal Values and Prediction Errors , 2008, The Journal of Neuroscience.

[44]  P. Dayan,et al.  Decision theory, reinforcement learning, and the brain , 2008, Cognitive, affective & behavioral neuroscience.

[45]  Timothy E. J. Behrens,et al.  Frontal Cortex Subregions Play Distinct Roles in Choices between Actions and Stimuli , 2008, The Journal of Neuroscience.

[46]  M. Delgado,et al.  Regulating the expectation of reward via cognitive strategies , 2008, Nature Neuroscience.

[47]  Colin Camerer,et al.  Neuroeconomics: decision making and the brain , 2008 .

[48]  M. Frank,et al.  Instructional control of reinforcement learning: A behavioral and neurocomputational investigation , 2009, Brain Research.

[49]  Y. Niv Reinforcement learning in the brain , 2009 .

[50]  Jian Li,et al.  Neural responses to sanction threats in two-party economic exchange , 2009, Proceedings of the National Academy of Sciences.

[51]  Colin Camerer,et al.  Self-control in decision-making involves modulation of the vmPFC valuation system , 2009, NeuroImage.

[52]  Tobias Egner,et al.  Prefrontal cortex and cognitive control: motivating functional hierarchies , 2009, Nature Neuroscience.

[53]  E. Koechlin,et al.  Motivation and cognitive control in the human prefrontal cortex , 2009, Nature Neuroscience.

[54]  Richard Gonzalez,et al.  Computational Models for the Combination of Advice and Individual Learning , 2009, Cogn. Sci..

[55]  P. Dayan,et al.  States versus Rewards: Dissociable Neural Prediction Error Signals Underlying Model-Based and Model-Free Reinforcement Learning , 2010, Neuron.

[56]  W. Schultz,et al.  Neural mechanisms of observational learning , 2010, Proceedings of the National Academy of Sciences.