Microstimulation of the Human Substantia Nigra Alters Reinforcement Learning

Animal studies have shown that substantia nigra (SN) dopaminergic (DA) neurons strengthen action–reward associations during reinforcement learning, but their role in human learning is not known. Here, we applied microstimulation in the SN of 11 patients undergoing deep brain stimulation surgery for the treatment of Parkinson's disease as they performed a two-alternative probability learning task in which rewards were contingent on stimuli, rather than actions. Subjects demonstrated decreased learning from reward trials that were accompanied by phasic SN microstimulation compared with reward trials without stimulation. Subjects who showed large decreases in learning also showed an increased bias toward repeating actions after stimulation trials; therefore, stimulation may have decreased learning by strengthening action–reward associations rather than stimulus–reward associations. Our findings build on previous studies implicating SN DA neurons in preferentially strengthening action–reward associations during reinforcement learning.

[1]  A. Grace,et al.  Are you or aren’t you? Challenges associated with physiologically identifying dopamine neurons , 2012, Trends in Neurosciences.

[2]  B. Rossion,et al.  Revisiting Snodgrass and Vanderwart's Object Pictorial Set: The Role of Surface Detail in Basic-Level Object Recognition , 2004, Perception.

[3]  J. Glowinski,et al.  Electrical Synapses between Dopaminergic Neurons of the Substantia Nigra Pars Compacta , 2005, The Journal of Neuroscience.

[4]  Matthew B. Stern,et al.  Bilateral Stimulation of the Subthalamic Nucleus in Parkinson’s Disease: Surgical Efficacy and Prediction of Outcome , 2004, Stereotactic and Functional Neurosurgery.

[5]  Richard S. Sutton,et al.  Time-Derivative Models of Pavlovian Reinforcement , 1990 .

[6]  H. Akaike A new look at the statistical model identification , 1974 .

[7]  R. Reid,et al.  Direct Activation of Sparse, Distributed Populations of Cortical Neurons by Electrical Microstimulation , 2009, Neuron.

[8]  C. Koch,et al.  Invariant visual representation by single neurons in the human brain , 2005, Nature.

[9]  D. Shohamy,et al.  A Role for the Medial Temporal Lobe in Feedback-Driven Learning: Evidence from Amnesia , 2013, The Journal of Neuroscience.

[10]  Sham M. Kakade,et al.  Opponent interactions between serotonin and dopamine , 2002, Neural Networks.

[11]  M. Gluck,et al.  Dopaminergic Drugs Modulate Learning Rates and Perseveration in Parkinson's Patients in a Dynamic Foraging Task , 2009, The Journal of Neuroscience.

[12]  T. Robbins,et al.  Enhanced or impaired cognitive function in Parkinson's disease as a function of dopaminergic medication and task demands. , 2001, Cerebral cortex.

[13]  J. Wickens,et al.  A cellular mechanism of reward-related learning , 2001, Nature.

[14]  Jian Liu,et al.  Changes in firing rate and pattern of GABAergic neurons in subregions of the substantia nigra pars reticulata in rat models of Parkinson's disease , 2010, Brain Research.

[15]  J. Hollerman,et al.  The effects of dopamine-depleting brain lesions on the electrophysiological activity of rat substantia nigra dopamine neurons , 1990, Brain Research.

[16]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[17]  W. Newsome,et al.  Choosing the greater of two goods: neural currencies for valuation and decision making , 2005, Nature Reviews Neuroscience.

[18]  A. Graybiel,et al.  The substantia nigra of the human brain. II. Patterns of loss of dopamine-containing neurons in Parkinson's disease. , 1999, Brain : a journal of neurology.

[19]  M. Frank,et al.  Do substantia nigra dopaminergic neurons differentiate between reward and punishment? , 2009, Journal of molecular cell biology.

[20]  J. Rinne,et al.  A quantitative morphometrical study of neuron degeneration in the substantia nigra in Parkinson's disease , 1996, Journal of the Neurological Sciences.

[21]  P. Glimcher,et al.  Value Representations in the Primate Striatum during Matching Behavior , 2008, Neuron.

[22]  P. Glimcher Understanding dopamine and reinforcement learning: The dopamine reward prediction error hypothesis , 2011, Proceedings of the National Academy of Sciences.

[23]  Karl J. Friston,et al.  Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning , 2004, Science.

[24]  R. Dolan,et al.  Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans , 2006, Nature.

[25]  Kelsey L. Clark,et al.  Probing neural circuitry and function with electrical microstimulation , 2011, Proceedings of the Royal Society B: Biological Sciences.

[26]  P. Glimcher,et al.  Midbrain Dopamine Neurons Encode a Quantitative Reward Prediction Error Signal , 2005, Neuron.

[27]  P. Dayan,et al.  A framework for mesencephalic dopamine systems based on predictive Hebbian learning , 1996, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[28]  J. Dostrovsky,et al.  Microstimulation-induced inhibition as a tool to aid targeting the ventral border of the subthalamic nucleus. , 2009, Journal of neurosurgery.

[29]  R. Rosenfeld Nature , 2009, Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery.

[30]  Michael J. Frank,et al.  By Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism , 2004, Science.

[31]  O. Hikosaka,et al.  Two types of dopamine neuron distinctly convey positive and negative motivational signals , 2009, Nature.

[32]  K. Deisseroth,et al.  Phasic Firing in Dopaminergic Neurons Is Sufficient for Behavioral Conditioning , 2009, Science.

[33]  Nikolaus R. McFarland,et al.  Striatonigrostriatal Pathways in Primates Form an Ascending Spiral from the Shell to the Dorsolateral Striatum , 2000, The Journal of Neuroscience.

[34]  JM Tepper,et al.  GABAA receptor-mediated inhibition of rat substantia nigra dopaminergic neurons by pars reticulata projection neurons , 1995, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[35]  P. Dayan,et al.  Dopamine and performance in a reinforcement learning task: evidence from Parkinson's disease. , 2012 .

[36]  A. Graybiel,et al.  The substantia nigra of the human brain. I. Nigrosomes and the nigral matrix, a compartmental organization based on calbindin D(28K) immunohistochemistry. , 1999, Brain : a journal of neurology.

[37]  P. Glimcher,et al.  Statistics of midbrain dopamine neuron spike trains in the awake primate. , 2007, Journal of neurophysiology.

[38]  Jennifer A. Mangels,et al.  A Neostriatal Habit Learning System in Humans , 1996, Science.

[39]  P. Dayan,et al.  Cortical substrates for exploratory decisions in humans , 2006, Nature.

[40]  J. Dudman,et al.  Neural signals of extinction in the inhibitory microcircuit of the ventral midbrain , 2012, Nature Neuroscience.

[41]  K. Sakai,et al.  Reinforcement learning: computing the temporal difference of values via distinct corticostriatal pathways , 2012, Trends in Neurosciences.

[42]  P. Dayan,et al.  Tonic dopamine: opportunity costs and the control of response vigor , 2007, Psychopharmacology.

[43]  A. Grace,et al.  Compensations after lesions of central dopaminergic neurons: some clinical and basic implications , 1990, Trends in Neurosciences.

[44]  J. Bolam,et al.  Structural correlates of heterogeneous in vivo activity of midbrain dopaminergic neurons , 2012, Nature Neuroscience.

[45]  M. Kahana,et al.  Human Substantia Nigra Neurons Encode Unexpected Financial Rewards , 2009, Science.

[46]  Michael J. Frank,et al.  Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning , 2007, Proceedings of the National Academy of Sciences.