Cerebral Cortex doi:10.1093/cercor/bhj004 Neural Coding of Distinct Statistical Properties of Reward Information in Humans

Brain processing of reward information is essential for complex functions such as learning and motivation. Recent primate electrophysiological studies using concepts from information, economic and learning theories indicate that the midbrain may code two statistical parameters of reward information: a transient reward error prediction signal that varies linearly with reward probability and a sustained signal that varies highly non-linearly with reward probability and that is highest with maximal reward uncertainty (reward probability = 0.5). Here, using event-related functional magnetic resonance imaging, we disentangled these two signals in humans using a novel paradigm that systematically varied monetary reward probability, magnitude and expected reward value. The midbrain was activated both transiently with the error prediction signal and in a sustained fashion with reward uncertainty. Moreover, distinct activity dynamics were observed in post-synaptic midbrain projection sites: the prefrontal cortex responded to the transient error prediction signal while the ventral striatum covaried with the sustained reward uncertainty signal. These data suggest that the prefrontal cortex may generate the reward prediction while the ventral striatum may be involved in motivational processes that are useful when an organism needs to obtain more information about its environment. Our results indicate that distinct functional brain networks code different aspects of the statistical properties of reward information in humans.

[1]  J. Pearce,et al.  A model for Pavlovian learning: Variations in the effectiveness of conditioned but not of unconditioned stimuli. , 1980 .

[2]  E. G. Jones Cerebral Cortex , 1987, Cerebral Cortex.

[3]  H Nishijo,et al.  Single neuron responses in amygdala of alert monkey during complex sensory stimulation with affective significance , 1988, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[4]  W. Schultz,et al.  Neuronal activity in monkey striatum related to the expectation of predictable environmental events. , 1992, Journal of neurophysiology.

[5]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[6]  B. Richmond,et al.  Neuronal Signals in the Monkey Ventral Striatum Related to Progress through a Predictable Series of Trials , 1998, The Journal of Neuroscience.

[7]  G. Schoenbaum,et al.  Orbitofrontal cortex and basolateral amygdala encode expected outcomes during learning , 1998, Nature Neuroscience.

[8]  K. Berridge,et al.  What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience? , 1998, Brain Research Reviews.

[9]  Michael L. Platt,et al.  Neural correlates of decision variables in parietal cortex , 1999, Nature.

[10]  J. Changeux,et al.  Reward-dependent learning in neuronal networks for planning and decision making. , 2000, Progress in brain research.

[11]  C. Bradberry Acute and Chronic Dopamine Dynamics in a Nonhuman Primate Model of Recreational Cocaine Use , 2000, The Journal of Neuroscience.

[12]  B. Mellers Choice and the relative pleasure of consequences. , 2000, Psychological bulletin.

[13]  W. Schultz Multiple reward signals in the brain , 2000, Nature Reviews Neuroscience.

[14]  Samuel M. McClure,et al.  Predictability Modulates Human Brain Response to Reward , 2001, The Journal of Neuroscience.

[15]  J. M. Anderson,et al.  Responses of human frontal cortex to surprising events are predicted by formal associative learning theory , 2001, Nature Neuroscience.

[16]  D. Kahneman,et al.  Functional Imaging of Neural Responses to Expectancy and Experience of Monetary Gains and Losses tasks with monetary payoffs , 2001 .

[17]  Keith Heberlein,et al.  Improved Shim by Subject Head Positioning , 2001 .

[18]  N. Logothetis,et al.  Neurophysiological investigation of the basis of the fMRI signal , 2001, Nature.

[19]  W. Schultz,et al.  Dopamine responses comply with basic assumptions of formal learning theory , 2001, Nature.

[20]  Brian Knutson,et al.  Anticipation of Increasing Monetary Reward Selectively Recruits Nucleus Accumbens , 2001, The Journal of Neuroscience.

[21]  Yves Burnod,et al.  An integrative theory of the phasic and tonic modes of dopamine modulation in the prefrontal cortex , 2002, Neural Networks.

[22]  Joshua W. Brown,et al.  Monitoring and Control of Action by the Frontal Lobes , 2002, Neuron.

[23]  J. O'Doherty,et al.  Neural Responses during Anticipation of a Primary Taste Reward , 2002, Neuron.

[24]  W. Schultz Getting Formal with Dopamine and Reward , 2002, Neuron.

[25]  P. Dayan,et al.  Reward, Motivation, and Reinforcement Learning , 2002, Neuron.

[26]  O. Hikosaka,et al.  Influence of reward expectation on visuospatial processing in macaque lateral prefrontal cortex. , 2002, Journal of neurophysiology.

[27]  Jordan Grafman,et al.  The Roles of Timing and Task Order during Task Switching , 2002, NeuroImage.

[28]  F. Tomasello,et al.  In vivo atlas of deep brain structures : with 3D reconstructions , 2002 .

[29]  Y. Burnod,et al.  A Model of Prefrontal Cortex Dopaminergic Modulation during the Delayed Alternation Task , 2002, Journal of Cognitive Neuroscience.

[30]  P. Montague,et al.  Activity in human ventral striatum locked to errors of reward prediction , 2002, Nature Neuroscience.

[31]  K. Hikosaka,et al.  Coding and Monitoring of Motivational Context in the Primate Prefrontal Cortex , 2002, The Journal of Neuroscience.

[32]  T. Robbins,et al.  Dopamine Release in the Dorsal Striatum during Cocaine-Seeking Behavior under the Control of a Drug-Associated Cue , 2002, The Journal of Neuroscience.

[33]  William J. Gehring,et al.  Medial prefrontal cortex and error potentials , 2002 .

[34]  S. Sesack,et al.  Anatomical Substrates for Glutamate‐Dopamine Interactions , 2003 .

[35]  E. Koechlin,et al.  The Architecture of Cognitive Control in the Human Prefrontal Cortex , 2003, Science.

[36]  Samuel M. McClure,et al.  Temporal Prediction Errors in a Passive Learning Task Activate Human Striatum , 2003, Neuron.

[37]  Karl J. Friston,et al.  Temporal Difference Models and Reward-Related Learning in the Human Brain , 2003, Neuron.

[38]  S. Sesack,et al.  Anatomical substrates for glutamate-dopamine interactions: evidence for specificity of connections and extrasynaptic actions. , 2003, Annals of the New York Academy of Sciences.

[39]  W. Schultz,et al.  Discrete Coding of Reward Probability and Uncertainty by Dopamine Neurons , 2003, Science.

[40]  P. Glimcher The neurobiology of visual-saccadic decision making. , 2003, Annual review of neuroscience.

[41]  Tatsuo K Sato,et al.  Correlated Coding of Motivation and Outcome of Decision by Dopamine Neurons , 2003, The Journal of Neuroscience.

[42]  M. Gluck,et al.  Human midbrain sensitivity to cognitive feedback and uncertainty during classification learning. , 2004, Journal of neurophysiology.

[43]  A. Dickinson,et al.  Prediction Error during Retrospective Revaluation of Causal Associations in Humans fMRI Evidence in Favor of an Associative Model of Learning , 2004, Neuron.

[44]  Clay B. Holroyd,et al.  Dorsal anterior cingulate cortex shows fMRI response to internal and external error signals , 2004, Nature Neuroscience.

[45]  Karl J. Friston,et al.  Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning , 2004, Science.

[46]  Rebecca Elliott,et al.  Instrumental responding for rewards is associated with enhanced neuronal response in subcortical reward systems , 2004, NeuroImage.

[47]  A. Yuille,et al.  Object perception as Bayesian inference. , 2004, Annual review of psychology.

[48]  R. E. Passingham,et al.  Prediction error for free monetary reward in the human prefrontal cortex , 2004, NeuroImage.

[49]  M. Delgado,et al.  Modulation of Caudate Activity by Action Contingency , 2004, Neuron.

[50]  Morten L Kringelbach,et al.  Taste-related activity in the human dorsolateral prefrontal cortex , 2004, NeuroImage.

[51]  Konrad Paul Kording,et al.  Bayesian integration in sensorimotor learning , 2004, Nature.

[52]  Martin P Paulus,et al.  Trend detection via temporal difference model predicts inferior prefrontal cortex activation during acquisition of advantageous action selection , 2004, NeuroImage.

[53]  Yael Niv,et al.  Uncertainty-based competition between prefrontal and striatal systems for behavioural control , 2005 .

[54]  J. Salamone,et al.  Beyond the reward hypothesis: alternative functions of nucleus accumbens dopamine. , 2005, Current opinion in pharmacology.

[55]  P. Dayan,et al.  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control , 2005, Nature Neuroscience.

[56]  P. Dayan,et al.  Dopamine, uncertainty and TD learning , 2005, Behavioral and Brain Functions.

[57]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .