Mesolimbic Dopamine Signals the Value of Work

Dopamine cell firing can encode errors in reward prediction, providing a learning signal to guide future behavior. Yet dopamine is also a key modulator of motivation, invigorating current behavior. Existing theories propose that fast (phasic) dopamine fluctuations support learning, whereas much slower (tonic) dopamine changes are involved in motivation. We examined dopamine release in the nucleus accumbens across multiple time scales, using complementary microdialysis and voltammetric methods during adaptive decision-making. We found that minute-by-minute dopamine levels covaried with reward rate and motivational vigor. Second-by-second dopamine release encoded an estimate of temporally discounted future reward (a value function). Changing dopamine immediately altered willingness to work and reinforced preceding action choices by encoding temporal-difference reward prediction errors. Our results indicate that dopamine conveys a single, rapidly evolving decision variable, the available reward for investment of effort, which is employed for both learning and motivational functions.

[1]  C. L. Hull The goal-gradient hypothesis and maze learning. , 1932 .

[2]  M S Buchsbaum,et al.  Dextroamphetamine. Its cognitive and behavioral effects in normal and hyperactive boys and normal men. , 1980, Archives of general psychiatry.

[3]  J. E. Mazur Tests of an equivalence rule for fixed and variable reinforcer delays. , 1984 .

[4]  B. Yamamoto,et al.  Regional brain dopamine metabolism: a marker for the speed, direction, and posture of moving animals. , 1985, Science.

[5]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[6]  A. Kacelnik Normative and descriptive models of decision making: time discounting and risk sensitivity. , 2007, Ciba Foundation symposium.

[7]  S. Ikemoto,et al.  The role of nucleus accumbens dopamine in motivated behavior: a unifying interpretation with special reference to reward-seeking , 1999, Brain Research Reviews.

[8]  J. Wickens,et al.  A cellular mechanism of reward-related learning , 2001, Nature.

[9]  R. Wightman,et al.  Response times of carbon fiber microelectrodes to dynamic changes in catecholamine concentration. , 2002, Analytical chemistry.

[10]  Sham M. Kakade,et al.  Opponent interactions between serotonin and dopamine , 2002, Neural Networks.

[11]  R. Palmiter,et al.  Reward without Dopamine , 2003, The Journal of Neuroscience.

[12]  Samuel M. McClure,et al.  A computational substrate for incentive salience , 2003, Trends in Neurosciences.

[13]  R. Wightman,et al.  Subsecond dopamine release promotes cocaine seeking , 2003, Nature.

[14]  W. Schultz,et al.  Discrete Coding of Reward Probability and Uncertainty by Dopamine Neurons , 2003, Science.

[15]  Tatsuo K Sato,et al.  Correlated Coding of Motivation and Outcome of Decision by Dopamine Neurons , 2003, The Journal of Neuroscience.

[16]  W. Newsome,et al.  Matching Behavior and the Representation of Value in the Parietal Cortex , 2004, Science.

[17]  R. Wightman,et al.  Dopamine Operates as a Subsecond Modulator of Food Seeking , 2004, The Journal of Neuroscience.

[18]  J. Salamone,et al.  Accumbens dopamine and the regulation of effort in food-seeking behavior: modulation of work output by different ratio or force requirements , 2004, Behavioural Brain Research.

[19]  K. Doya,et al.  Representation of Action-Specific Reward Values in the Striatum , 2005, Science.

[20]  R. Wightman,et al.  Real-time measurement of dopamine fluctuations after cocaine in the brain of behaving rats. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[21]  George Ainslie,et al.  Précis of Breakdown of Will , 2005, Behavioral and Brain Sciences.

[22]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[23]  Peter Dayan,et al.  How fast to work: Response vigor, motivation and tonic dopamine , 2005, NIPS.

[24]  P. Glimcher,et al.  JOURNAL OF THE EXPERIMENTAL ANALYSIS OF BEHAVIOR 2005, 84, 555–579 NUMBER 3(NOVEMBER) DYNAMIC RESPONSE-BY-RESPONSE MODELS OF MATCHING BEHAVIOR IN RHESUS MONKEYS , 2022 .

[25]  K. Berridge The debate over dopamine’s role in reward: the case for incentive salience , 2007, Psychopharmacology.

[26]  Philip Holmes,et al.  Rapid decision threshold modulation by reward rate in a neural network , 2006, Neural Networks.

[27]  P. Balsam,et al.  Mice with Chronically Elevated Dopamine Exhibit Enhanced Motivation, but not Learning, for a Food Reward , 2006, Neuropsychopharmacology.

[28]  E. Vaadia,et al.  Midbrain dopamine neurons encode decisions for future action , 2006, Nature Neuroscience.

[29]  David S. Touretzky,et al.  Representation and Timing in Theories of the Dopamine System , 2006, Neural Computation.

[30]  R. Wightman,et al.  Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens , 2007, Nature Neuroscience.

[31]  P. Glimcher,et al.  The neural correlates of subjective value during intertemporal choice , 2007, Nature Neuroscience.

[32]  A. Baron,et al.  What 50 years of research tell us about pausing under ratio schedules of reinforcement , 2008, The Behavior analyst.

[33]  W. Schultz,et al.  Influence of Reward Delays on Responses of Dopamine Neurons , 2008, The Journal of Neuroscience.

[34]  M. Roitman,et al.  Regional specificity in the real‐time development of phasic dopamine transmission patterns during acquisition of a cue–cocaine association in rats , 2009, The European journal of neuroscience.

[35]  O. Hikosaka,et al.  Two types of dopamine neuron distinctly convey positive and negative motivational signals , 2009, Nature.

[36]  M. Walton,et al.  Dissociable cost and benefit encoding of future rewards by mesolimbic dopamine , 2009, Nature Neuroscience.

[37]  S. Nicola The Flexible Approach Hypothesis: Unification of Effort and Cue-Responding Hypotheses for the Role of Nucleus Accumbens Dopamine in the Activation of Reward-Seeking Behavior , 2010, The Journal of Neuroscience.

[38]  M. Desmurget,et al.  Basal ganglia contributions to motor control: a vigorous tutor , 2010, Current Opinion in Neurobiology.

[39]  Alexander B. Wiltschko,et al.  Selective Activation of Striatal Fast-Spiking Interneurons during Choice Execution , 2010, Neuron.

[40]  Rune W. Berg,et al.  Influence of Phasic and Tonic Dopamine Release on Receptor Activation , 2010, The Journal of Neuroscience.

[41]  Ilana B. Witten,et al.  Recombinase-Driver Rat Lines: Tools, Techniques, and Optogenetic Application to Dopamine-Mediated Reinforcement , 2011, Neuron.

[42]  Peter Dayan,et al.  Vigor in the Face of Fluctuating Rates of Reward: An Experimental Examination , 2011, Journal of Cognitive Neuroscience.

[43]  K. Deisseroth,et al.  Optogenetic Interrogation of Dopaminergic Modulation of the Multiple Phases of Reward-Seeking Behavior , 2011, The Journal of Neuroscience.

[44]  Matthijs A. A. van der Meer,et al.  Ventral striatum: a critical look at models of learning and evaluation , 2011, Current Opinion in Neurobiology.

[45]  H. de Wit,et al.  Amping Up Effort: Effects of d-Amphetamine on Human Effort-Based Decision-Making , 2011, The Journal of Neuroscience.

[46]  O. Mabrouk,et al.  Mass spectrometry "sensor" for in vivo acetylcholine monitoring. , 2012, Analytical chemistry.

[47]  P. Cisek,et al.  From anticipation to action, the role of dopamine in perceptual decision making: an fMRI-tyrosine depletion study. , 2012, Journal of neurophysiology.

[48]  O. Hikosaka,et al.  The Primate Ventral Pallidum Encodes Expected Reward Value and Regulates Motor Action , 2012, Neuron.

[49]  K. Deisseroth,et al.  Striatal Dopamine Release Is Triggered by Synchronized Activity in Cholinergic Interneurons , 2012, Neuron.

[50]  O. Mabrouk,et al.  In vivo neurochemical monitoring using benzoyl chloride derivatization and liquid chromatography-mass spectrometry. , 2012, Analytical chemistry.

[51]  Thomas R. Reppert,et al.  Evidence for Hyperbolic Temporal Discounting of Reward in Control of Movements , 2012, The Journal of Neuroscience.

[52]  J. Salamone,et al.  The Mysterious Motivational Functions of Mesolimbic Dopamine , 2012, Neuron.

[53]  Daniel K. Leventhal,et al.  Basal Ganglia Beta Oscillations Accompany Cue Utilization , 2012, Neuron.

[54]  X. Zhuang,et al.  Putting desire on a budget: dopamine and energy expenditure, reconciling reward and resources , 2012, Front. Integr. Neurosci..

[55]  R. Wightman,et al.  Optimizing the Temporal Resolution of Fast-Scan Cyclic Voltammetry. , 2012, ACS chemical neuroscience.

[56]  M. Khamassi,et al.  Dopaminergic Control of the Exploration-Exploitation Trade-Off via the Basal Ganglia , 2012, Front. Neurosci..

[57]  C. Fiorillo,et al.  Optogenetic Mimicry of the Transient Activation of Dopamine Neurons by Natural Reward Is Sufficient for Operant Reinforcement , 2012, PloS one.

[58]  N. Uchida,et al.  The dorsomedial striatum encodes net expected return, critical for energizing performance vigor , 2013, Nature Neuroscience.

[59]  Ulrik R Beierholm,et al.  Dopamine Modulates Reward-Related Vigor , 2013, Neuropsychopharmacology.

[60]  A. Graybiel,et al.  Prolonged Dopamine Signalling in Striatum Signals Proximity and Value of Distant Rewards , 2013, Nature.

[61]  Josiah R. Boivin,et al.  A Causal Link Between Prediction Errors, Dopamine Neurons and Learning , 2013, Nature Neuroscience.

[62]  Daniel K. Leventhal,et al.  Canceling actions involves a race between basal ganglia pathways , 2013, Nature Neuroscience.

[63]  P. Glimcher,et al.  Phasic Dopamine Release in the Rat Nucleus Accumbens Symmetrically Encodes a Reward Prediction Error Term , 2014, The Journal of Neuroscience.

[64]  Peter Dayan,et al.  Optimal indolence: a normative microscopic approach to work and leisure , 2014, Journal of The Royal Society Interface.

[65]  Samuel Gershman,et al.  Dopamine Ramps Are a Consequence of Reward Prediction Errors , 2014, Neural Computation.

[66]  Daniel K. Leventhal,et al.  Dissociable effects of dopamine on learning and performance within sensorimotor striatum. , 2014, Basal ganglia.

[67]  Kenji Morita,et al.  Striatal dopamine ramping may indicate flexible reinforcement learning with forgetting in the cortico-basal ganglia circuits , 2014, Front. Neural Circuits.

[68]  S. Nicola,et al.  Dopamine Invigorates Reward Seeking by Promoting Cue-Evoked Excitation in the Nucleus Accumbens , 2014, The Journal of Neuroscience.