Midbrain dopamine neurons encode decisions for future action

Current models of the basal ganglia and dopamine neurons emphasize their role in reinforcement learning. However, the role of dopamine neurons in decision making is still unclear. We recorded from dopamine neurons in monkeys engaged in two types of trial: reference trials in an instructed-choice task and decision trials in a two-armed bandit decision task. We show that the activity of dopamine neurons in the decision setting is modulated according to the value of the upcoming action. Moreover, analysis of the probability matching strategy in the decision trials revealed that the dopamine population activity and not the reward during reference trials determines choice behavior. Because dopamine neurons do not have spatial or motor properties, we conclude that immediate decisions are likely to be generated elsewhere and conveyed to the dopamine neurons, which play a role in shaping long-term decision policy through dynamic modulation of the efficacy of basal ganglia synapses.

[1]  R. Herrnstein.  On the law of effect. , 1970, Journal of the experimental analysis of behavior.

[2]  A. Tversky,et al.  The framing of decisions and the psychology of choice. , 1981, Science.

[3]  A. Grace,et al.  Intracellular and extracellular electrophysiology of nigral dopaminergic neurons—1. Identification and characterization , 1983, Neuroscience.

[4]  W. Cowan,et al.  A stereotaxic atlas of the brain of the cynomolgus monkey (Macaca fascicularis) , 1984, The Journal of comparative neurology.

[5]  Mahesan Niranjan,et al.  On-line Q-learning using connectionist systems , 1994 .

[6]  O. Hikosaka.  Models of information processing in the basal Ganglia edited by James C. Houk, Joel L. Davis and David G. Beiser, The MIT Press, 1995. $60.00 (400 pp) ISBN 0 262 08234 9 , 1995, Trends in Neurosciences.

[7]  A. Barto,et al.  Adaptive Critics and the Basal Ganglia , 1994 .

[8]  J. Mink.  THE BASAL GANGLIA: FOCUSED SELECTION AND INHIBITION OF COMPETING MOTOR PROGRAMS , 1996, Progress in Neurobiology.

[9]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[10]  J. Hollerman,et al.  Dopamine neurons report an error in the temporal prediction of reward during learning , 1998, Nature Neuroscience.

[11]  Nir Vulkan.  An Economist's Perspective on Probability Matching , 2000 .

[12]  Richard F. Martin,et al.  Primate brain maps : structure of the macaque brain , 2000 .

[13]  Michael B. Miller,et al.  The Left Hemisphere's Role in Hypothesis Formation , 2000, The Journal of Neuroscience.

[14]  J. Wickens,et al.  A cellular mechanism of reward-related learning , 2001, Nature.

[15]  W. Schultz,et al.  Dopamine responses comply with basic assumptions of formal learning theory , 2001, Nature.

[16]  P. Dayan,et al.  Reward, Motivation, and Reinforcement Learning , 2002, Neuron.

[17]  P. Montague,et al.  Activity in human ventral striatum locked to errors of reward prediction , 2002, Nature Neuroscience.

[18]  Samuel M. McClure,et al.  A computational substrate for incentive salience , 2003, Trends in Neurosciences.

[19]  W. Schultz,et al.  Discrete Coding of Reward Probability and Uncertainty by Dopamine Neurons , 2003, Science.

[20]  Tatsuo K Sato,et al.  Correlated Coding of Motivation and Outcome of Decision by Dopamine Neurons , 2003, The Journal of Neuroscience.

[21]  H. Bergman,et al.  Information processing, dimensionality reduction and reinforcement learning in the basal ganglia , 2003, Progress in Neurobiology.

[22]  R. Wise.  Dopamine, learning and motivation , 2004, Nature Reviews Neuroscience.

[23]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[24]  J. Bolam,et al.  Uniform Inhibition of Dopamine Neurons in the Ventral Tegmental Area by Aversive Stimuli , 2004, Science.

[25]  O. Hikosaka,et al.  Dopamine Neurons Can Represent Context-Dependent Prediction Error , 2004, Neuron.

[26]  R. Romo,et al.  Neuronal Correlates of a Perceptual Decision in Ventral Premotor Cortex , 2004, Neuron.

[27]  W. Newsome,et al.  Matching Behavior and the Representation of Value in the Parietal Cortex , 2004, Science.

[28]  J. Wickens,et al.  Computational models of the basal ganglia: from robots to membranes , 2004, Trends in Neurosciences.

[29]  Peter Dayan,et al.  Technical Note: Q-Learning , 2004, Machine Learning.

[30]  Hagai Bergman,et al.  Discharge rate of substantia nigra pars reticulata neurons is reduced in non-parkinsonian monkeys with apomorphine-induced orofacial dyskinesia. , 2004, Journal of Neurophysiology.

[31]  S. Cragg,et al.  DAncing past the DAT at a DA synapse , 2004, Trends in Neurosciences.

[32]  D. Barraclough,et al.  Prefrontal cortex and decision making in a mixed-strategy game , 2004, Nature Neuroscience.

[33]  E. Vaadia,et al.  Coincident but Distinct Messages of Midbrain Dopamine and Striatal Tonically Active Neurons , 2004, Neuron.

[34]  W. Pan,et al.  Dopamine Cells Respond to Predicted Events during Classical Conditioning: Evidence for Eligibility Traces in the Reward-Learning Network , 2005, The Journal of Neuroscience.

[35]  K. Doya,et al.  Representation of Action-Specific Reward Values in the Striatum , 2005, Science.

[36]  J. Wickens,et al.  Chapter IV Structural and functional interactions in the striatum at the receptor level , 2005 .

[37]  W. Schultz,et al.  Adaptive Coding of Reward Value by Dopamine Neurons , 2005, Science.

[38]  J. Mayhew,et al.  How Visual Stimuli Activate Dopaminergic Neurons at Short Latency , 2005, Science.

[39]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[40]  P. Glimcher,et al.  Midbrain Dopamine Neurons Encode a Quantitative Reward Prediction Error Signal , 2005, Neuron.

[41]  P. Glimcher,et al.  JOURNAL OF THE EXPERIMENTAL ANALYSIS OF BEHAVIOR 2005, 84, 555–579 NUMBER 3(NOVEMBER) DYNAMIC RESPONSE-BY-RESPONSE MODELS OF MATCHING BEHAVIOR IN RHESUS MONKEYS , 2022 .

[42]  R. K. Simpson.  Nature Neuroscience , 2022 .