Dopamine, reinforcement learning, and addiction.

Dopamine is intimately linked with the modes of action of drugs of addiction. However, although its role in the initiation of drug abuse seems relatively uncomplicated, its possible involvement in the development of compulsive drug taking, and indeed vulnerability and relapse, is less clear. We first describe a modern reinforcement learning view of affective control, focusing on the roles for dopamine. We then use this as a framework to sketch various notions of the neuromodulator's possible participation in initiation and compulsion. We end with some pointers towards future theoretical developments.

[1]  O. Mowrer On the dual nature of learning—a re-interpretation of "conditioning" and "problem-solving." , 1947 .

[2]  K. Breland,et al.  The misbehavior of organisms. , 1961 .

[3]  B. Bernstein,et al.  Animal Behavior , 1927, Japanese Marine Life.

[4]  L. Kamin Predictability, surprise, attention, and conditioning , 1967 .

[5]  D. R. Williams,et al.  Auto-maintenance in the pigeon: sustained pecking despite contingent non-reinforcement. , 1969, Journal of the experimental analysis of behavior.

[6]  R. Rescorla,et al.  A theory of Pavlovian conditioning : Variations in the effectiveness of reinforcement and nonreinforcement , 1972 .

[7]  R. Solomon,et al.  An opponent-process theory of motivation. I. Temporal dynamics of affect. , 1974, Psychological review.

[8]  F. Bloom,et al.  Locomotor activation induced by infusion of endorphins into the ventral tegmental area: evidence for opiate-dopamine interactions. , 1980, Proceedings of the National Academy of Sciences of the United States of America.

[9]  S. Grossberg Processing of expected and unexpected events during conditioning and attention: a psychophysiological theory. , 1982, Psychological review.

[10]  F. Masterson,et al.  Species-specific defense reactions and avoidance learning , 1982, The Pavlovian Journal of Biological Science.

[11]  H. Barlow Vision: A computational investigation into the human representation and processing of visual information: David Marr. San Francisco: W. H. Freeman, 1982. pp. xvi + 397 , 1983 .

[12]  S. Grossberg Some normal and abnormal behavioral syndromes due to transmitter gating of opponent processes. , 1984, Biological psychiatry.

[13]  W. Hershberger An approach through the looking-glass , 1986 .

[14]  C. Gallistel The role of the dopaminergic projections in MFB self-stimulation , 1986, Behavioural Brain Research.

[15]  R. Wise,et al.  A psychomotor stimulant theory of addiction. , 1987, Psychological review.

[16]  D. Blanchard,et al.  Ethoexperimental approaches to the biology of emotion. , 1988, Annual review of psychology.

[17]  G. Di Chiara,et al.  Drugs abused by humans preferentially increase synaptic dopamine concentrations in the mesolimbic system of freely moving rats. , 1988, Proceedings of the National Academy of Sciences of the United States of America.

[18]  R. Wise Opiate reward: Sites and substrates , 1989, Neuroscience & Biobehavioral Reviews.

[19]  P. Mason,et al.  Neurotransmitters in nociceptive modulatory circuits. , 1991, Annual review of neuroscience.

[20]  R. Spanagel,et al.  Opposing tonically active endogenous opioid systems modulate the mesolimbic dopaminergic pathway. , 1992, Proceedings of the National Academy of Sciences of the United States of America.

[21]  M. Le Moal,et al.  Interaction between Endogenous Opioids and Dopamine within the Nucleus Accumbens a , 1992, Annals of the New York Academy of Sciences.

[22]  G. Koob Drugs of abuse: anatomy, pharmacology and function of reward pathways. , 1992, Trends in pharmacological sciences.

[23]  Karl J. Friston,et al.  Value-dependent selection in the brain: Simulation in a synthetic neural model , 1994, Neuroscience.

[24]  Ben J. A. Kröse,et al.  Learning from delayed rewards , 1995, Robotics Auton. Syst..

[25]  A. Barto,et al.  Adaptive Critics and the Basal Ganglia , 1994 .

[26]  K. Berridge,et al.  Central enhancement of taste pleasure by intraventricular morphine. , 1995, Neurobiology.

[27]  P. Dayan,et al.  A framework for mesencephalic dopamine systems based on predictive Hebbian learning , 1996, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[28]  J. Horvitz,et al.  Burst activity of ventral tegmental dopamine neurons is elicited by sensory stimuli in the awake cat , 1997, Brain Research.

[29]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[30]  Nestor A. Schmajuk,et al.  Escape, Avoidance, and Imitation: A Neural Network Approach , 1997, Adapt. Behav..

[31]  G F Koob,et al.  Transition from moderate to excessive drug intake: change in hedonic set point. , 1998, Science.

[32]  T. Tzschentke,et al.  Measuring reward with the conditioned place preference paradigm: a comprehensive review of drug effects, recent progress and new issues , 1998, Progress in Neurobiology.

[33]  F. Weiss,et al.  The dopamine hypothesis of reward: past and current status , 1999, Trends in Neurosciences.

[34]  W. Schultz,et al.  A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task , 1999, Neuroscience.

[35]  J. Mirenowicz,et al.  Dissociation of Pavlovian and instrumental incentive learning under dopamine antagonists. , 2000, Behavioral neuroscience.

[36]  E. Stein,et al.  Cue-induced cocaine craving: neuroanatomical specificity for drug users and drug stimuli. , 2000, The American journal of psychiatry.

[37]  K. Berridge,et al.  Intra-Accumbens Amphetamine Increases the Conditioned Incentive Salience of Sucrose Reward: Enhancement of Reward “Wanting” without Enhanced “Liking” or Response Reinforcement , 2000, The Journal of Neuroscience.

[38]  D. Joel,et al.  The connections of the dopaminergic system with the striatum in rats and primates: an analysis with respect to the functional and compartmental organization of the striatum , 2000, Neuroscience.

[39]  Nikolaus R. McFarland,et al.  Striatonigrostriatal Pathways in Primates Form an Ascending Spiral from the Shell to the Dorsolateral Striatum , 2000, The Journal of Neuroscience.

[40]  J. Wickens,et al.  A cellular mechanism of reward-related learning , 2001, Nature.

[41]  P. Kalivas,et al.  The Circuitry Mediating Cocaine-Induced Reinstatement of Drug-Seeking Behavior , 2001, The Journal of Neuroscience.

[42]  Wei Li,et al.  A Computational Model of Learned Avoidance Behavior in a One-Way Avoidance Experiment , 2001, Adapt. Behav..

[43]  K. Berridge,et al.  Incentive-sensitization and addiction. , 2001, Addiction.

[44]  K. Berridge,et al.  The Neuroscience of Natural Rewards: Relevance to Addictive Drugs , 2002, The Journal of Neuroscience.

[45]  Sham M. Kakade,et al.  Opponent interactions between serotonin and dopamine , 2002, Neural Networks.

[46]  W. Schultz Getting Formal with Dopamine and Reward , 2002, Neuron.

[47]  P. Dayan,et al.  Reward, Motivation, and Reinforcement Learning , 2002, Neuron.

[48]  M. El-Sabaawi Breakdown of Will , 2002 .

[49]  T. Robinson,et al.  Memory Processes Governing Amphetamine-induced Psychomotor Sensitization , 2002, Neuropsychopharmacology.

[50]  Peter Dayan,et al.  Dopamine: generalization and bonuses , 2002, Neural Networks.

[51]  G. Chiara Nucleus accumbens shell and core dopamine: differential role in behavior and addiction , 2002, Behavioural Brain Research.

[52]  K. Berridge,et al.  Positive and Negative Motivation in Nucleus Accumbens Shell: Bivalent Rostrocaudal Gradients for GABA-Elicited Eating, Taste “Liking”/“Disliking” Reactions, Place Preference/Avoidance, and Fear , 2002, The Journal of Neuroscience.

[53]  J. Wickens,et al.  Neural mechanisms of reward-related motor learning , 2003, Current Opinion in Neurobiology.

[54]  J. Salamone,et al.  Nucleus Accumbens Dopamine and the Regulation of Effort in Food-Seeking Behavior: Implications for Studies of Natural Motivation, Psychiatry, and Drug Abuse , 2003, Journal of Pharmacology and Experimental Therapeutics.

[55]  Samuel M. McClure,et al.  A computational substrate for incentive salience , 2003, Trends in Neurosciences.

[56]  K. Berridge,et al.  Glutamate motivational ensembles in nucleus accumbens: rostrocaudal shell gradients of fear and feeding , 2003, The European journal of neuroscience.

[57]  Kent C. Berridge,et al.  Pleasures of the brain , 2003, Brain and Cognition.

[58]  R. Wightman,et al.  Subsecond dopamine release promotes cocaine seeking , 2003, Nature.

[59]  A. Nieoullon,et al.  Dopamine: a key regulator to adapt action, emotion, motivation and cognition , 2003, Current opinion in neurology.

[60]  Tatsuo K Sato,et al.  Correlated Coding of Motivation and Outcome of Decision by Dopamine Neurons , 2003, The Journal of Neuroscience.

[61]  N. Daw,et al.  Reinforcement learning models of the dopamine system and their behavioral implications , 2003 .

[62]  P. Kalivas,et al.  Brain circuitry and the reinstatement of cocaine-seeking behavior , 2003, Psychopharmacology.

[63]  P. Corr,et al.  A two-dimensional neuropsychology of defense: fear/anxiety and defensive distance , 2004, Neuroscience & Biobehavioral Reviews.

[64]  Ronald J. Williams,et al.  Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning , 2004, Machine Learning.

[65]  R. A. Fuchs,et al.  Differential involvement of the core and shell subregions of the nucleus accumbens in conditioned cue-induced reinstatement of cocaine seeking in rats , 2004, Psychopharmacology.

[66]  R. Wightman,et al.  Cannabinoids Enhance Subsecond Dopamine Release in the Nucleus Accumbens of Awake Rats , 2004, The Journal of Neuroscience.

[67]  Michael J. Frank,et al.  By Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism , 2004, Science.

[68]  B. Everitt,et al.  Neural and psychological mechanisms underlying appetitive learning: links to drug addiction , 2004, Current Opinion in Neurobiology.

[69]  Mathias Schreckenberger,et al.  Correlation between dopamine D(2) receptors in the ventral striatum and central processing of alcohol cues and craving. , 2004, The American journal of psychiatry.

[70]  J. Swanson,et al.  Dopamine in drug abuse and addiction: results from imaging studies and treatment implications , 2004, Molecular Psychiatry.

[71]  A. Redish,et al.  Addiction as a Computational Process Gone Awry , 2004, Science.

[72]  Paul Cumming,et al.  Correlation of alcohol craving with striatal dopamine synthesis capacity and D2/3 receptor availability: a combined [18F]DOPA and [18F]DMFP PET study in detoxified alcoholic patients. , 2005, The American journal of psychiatry.

[73]  R. Wightman,et al.  Real-time measurement of dopamine fluctuations after cocaine in the brain of behaving rats. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[74]  Angela J. Yu,et al.  Uncertainty, Neuromodulation, and Attention , 2005, Neuron.

[75]  T. Robbins,et al.  Neural systems of reinforcement for drug addiction: from actions to habits to compulsion , 2005, Nature Neuroscience.

[76]  R. Wise Forebrain substrates of reward and motivation , 2005, The Journal of comparative neurology.

[77]  P. Dayan,et al.  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control , 2005, Nature Neuroscience.

[78]  J. Wickens,et al.  Striatal dopamine in motor activation and reward-mediated learning: steps towards a unifying model , 2005, Journal of Neural Transmission / General Section JNT.

[79]  Michael J. Frank,et al.  Dynamic Dopamine Modulation in the Basal Ganglia: A Neurocomputational Account of Cognitive Deficits in Medicated and Nonmedicated Parkinsonism , 2005, Journal of Cognitive Neuroscience.

[80]  P. Dayan,et al.  Dopamine, learning, and impulsivity: a biological account of attention-deficit/hyperactivity disorder. , 2005, Journal of child and adolescent psychopharmacology.

[81]  B. Balleine Neural bases of food-seeking: Affect, arousal and reward in corticostriatolimbic circuits , 2005, Physiology & Behavior.

[82]  S. Hyman Addiction: a disease of learning and memory. , 2005, The American journal of psychiatry.

[83]  Richard S. Sutton,et al.  Learning to predict by the methods of temporal differences , 1988, Machine Learning.

[84]  R. Palmiter,et al.  Morphine reward in dopamine-deficient mice , 2005, Nature.

[85]  S. Hyman,et al.  Neural mechanisms of addiction: the role of reward-related learning and memory. , 2006, Annual review of neuroscience.

[86]  K. Berridge The debate over dopamine’s role in reward: the case for incentive salience , 2007, Psychopharmacology.

[87]  J. Salamone,et al.  Effort-related functions of nucleus accumbens dopamine and associated forebrain circuits , 2007, Psychopharmacology.

[88]  P. Dayan,et al.  Cortical substrates for exploratory decisions in humans , 2006, Nature.

[89]  R. Dolan,et al.  Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans , 2006, Nature.

[90]  Mitsuo Kawato,et al.  Heterarchical reinforcement-learning model for integration of multiple cortico-striatal loops: fMRI examination in stimulus-action-reward association learning , 2006, Neural Networks.

[91]  Peter Dayan,et al.  Non-commercial Research and Educational Use including without Limitation Use in Instruction at Your Institution, Sending It to Specific Colleagues That You Know, and Providing a Copy to Your Institution's Administrator. All Other Uses, Reproduction and Distribution, including without Limitation Comm , 2022 .

[92]  Michael J. Frank,et al.  Hold your horses: A dynamic computational role for the subthalamic nucleus in decision making , 2006, Neural Networks.

[93]  E. Vaadia,et al.  Midbrain dopamine neurons encode decisions for future action , 2006, Nature Neuroscience.

[94]  P. Shizgal,et al.  Prolonged rewarding stimulation of the rat medial forebrain bundle: neurochemical and behavioral consequences. , 2006, Behavioral neuroscience.

[95]  P. Dayan,et al.  Tonic dopamine: opportunity costs and the control of response vigor , 2007, Psychopharmacology.

[96]  L. Panlilio,et al.  Blocking of conditioning to a cocaine-paired stimulus: Testing the hypothesis that cocaine perpetually produces a signal of larger-than-expected reward , 2007, Pharmacology Biochemistry and Behavior.

[97]  A. Grace,et al.  The Yin and Yang of dopamine release: a new perspective , 2007, Neuropharmacology.

[98]  Trevor W Robbins,et al.  The Orbital Prefrontal Cortex and Drug Addiction in Laboratory Animals and Humans , 2007, Annals of the New York Academy of Sciences.

[99]  Jadin C. Jackson,et al.  Reconciling reinforcement learning models with behavioral extinction and renewal: implications for addiction, relapse, and problem gambling. , 2007, Psychological review.

[100]  S. Ikemoto Dopamine reward circuitry: Two projection systems from the ventral midbrain to the nucleus accumbens–olfactory tubercle complex , 2007, Brain Research Reviews.

[101]  M. Roesch,et al.  Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards , 2007, Nature Neuroscience.

[102]  R. See,et al.  The role of dorsal vs ventral striatal pathways in cocaine-seeking behavior after prolonged abstinence in rats , 2007, Psychopharmacology.

[103]  R. Wightman,et al.  Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens , 2007, Nature Neuroscience.

[104]  J. Krakauer,et al.  Why Don't We Move Faster? Parkinson's Disease, Movement Vigor, and Implicit Motivation , 2007, The Journal of Neuroscience.

[105]  Michael Moutoussis,et al.  Persecutory delusions and the conditioned avoidance paradigm: Towards an integration of the psychology and biology of paranoia , 2007, Cognitive neuropsychiatry.

[106]  Michael J. Frank,et al.  Hold Your Horses: Impulsivity, Deep Brain Stimulation, and Medication in Parkinsonism , 2007, Science.

[107]  Brian Knutson,et al.  Dysfunction of reward processing correlates with alcohol craving in detoxified alcoholics , 2007, NeuroImage.

[108]  Young T. Hong,et al.  Nucleus Accumbens D2/3 Receptors Predict Trait Impulsivity and Cocaine Reinforcement , 2007, Science.

[109]  Peter Dayan,et al.  The role of value systems in decision making. , 2008 .

[110]  R. Wightman,et al.  Dynamic changes in accumbens dopamine correlate with learning during intracranial self-stimulation , 2008, Proceedings of the National Academy of Sciences.

[111]  K. Berridge,et al.  The incentive sensitization theory of addiction: some current issues , 2008, Philosophical Transactions of the Royal Society B: Biological Sciences.

[112]  T. Robbins,et al.  Neural mechanisms underlying the vulnerability to develop compulsive drug-seeking habits and addiction , 2008, Philosophical Transactions of the Royal Society B: Biological Sciences.

[113]  P. Kalivas,et al.  Drug Addiction as a Pathology of Staged Neuroplasticity , 2008, Neuropsychopharmacology.

[114]  J. Stewart Psychological and neural mechanisms of relapse , 2008, Philosophical Transactions of the Royal Society B: Biological Sciences.

[115]  Peter Dayan,et al.  Serotonin, Inhibition, and Negative Mood , 2007, PLoS Comput. Biol..

[116]  P. Dayan,et al.  A temporal difference account of avoidance learning , 2008, Network.

[117]  K. Berridge,et al.  Emotional environments retune the valence of appetitive versus fearful functions in nucleus accumbens , 2008, Nature Neuroscience.

[118]  M. Milders,et al.  Abnormal Temporal Difference Reward-learning Signals in Major Depression Department of Radiology And , 2022 .

[119]  R. Wightman,et al.  Preferential Enhancement of Dopamine Transmission within the Nucleus Accumbens Shell by Cocaine Is Attributable to a Direct Increase in Phasic Dopamine Release Events , 2008, The Journal of Neuroscience.

[120]  M. Le Moal,et al.  Addiction and the brain antireward system. , 2008, Annual review of psychology.

[121]  Adam Johnson,et al.  Addiction as vulnerabilities in the decision process , 2008, Behavioral and Brain Sciences.

[122]  M. Le Moal,et al.  Neurobiological mechanisms for opponent motivational processes in addiction , 2008, Philosophical Transactions of the Royal Society B: Biological Sciences.

[123]  L. Clark,et al.  Impulsivity as a vulnerability marker for substance-use disorders: Review of findings from high-risk research, problem gamblers and genetic association studies , 2008, Neuroscience & Biobehavioral Reviews.

[124]  Huda Akil,et al.  Individual differences in the attribution of incentive salience to reward-related cues: Implications for addiction , 2009, Neuropharmacology.

[125]  Peter Dayan,et al.  Values and Actions in Aversion , 2009 .