Role of Striatum in Updating Values of Chosen Actions

The striatum is thought to play a crucial role in value-based decision making. Although a large body of evidence suggests its involvement in action selection as well as action evaluation, underlying neural processes for these functions of the striatum are largely unknown. To obtain insights on this matter, we simultaneously recorded neuronal activity in the dorsal and ventral striatum of rats performing a dynamic two-armed bandit task, and examined temporal profiles of neural signals related to animal's choice, its outcome, and action value. Whereas significant neural signals for action value were found in both structures before animal's choice of action, signals related to the upcoming choice were relatively weak and began to emerge only in the dorsal striatum ∼200 ms before the behavioral manifestation of the animal's choice. In contrast, once the animal revealed its choice, signals related to choice and its value increased steeply and persisted until the outcome of animal's choice was revealed, so that some neurons in both structures concurrently conveyed signals related to animal's choice, its outcome, and the value of chosen action. Thus, all the components necessary for updating values of chosen actions were available in the striatum. These results suggest that the striatum not only represents values associated with potential choices before animal's choice of action, but might also update the value of chosen action once its outcome is revealed. In contrast, action selection might take place elsewhere or in the dorsal striatum only immediately before its behavioral manifestation.

[1]  Michael H. Kutner Applied Linear Statistical Models , 1974 .

[2]  V. Barnett,et al.  Applied Linear Statistical Models , 1975 .

[3]  Douglas L. Jones,et al.  From motivation to action: Functional interface between the limbic system and the motor system , 1980, Progress in Neurobiology.

[4]  J. Penney,et al.  The functional anatomy of basal ganglia disorders , 1989, Trends in Neurosciences.

[5]  O. Hikosaka,et al.  Functional properties of monkey caudate neurons. I. Activities related to saccadic eye movements. , 1989, Journal of neurophysiology.

[6]  G. E. Alexander,et al.  Preparation for movement: neural representations of intended direction in three motor areas of the monkey. , 1990, Journal of neurophysiology.

[7]  G. E. Alexander,et al.  Functional architecture of basal ganglia circuits: neural substrates of parallel processing , 1990, Trends in Neurosciences.

[8]  W. Schultz,et al.  Neuronal activity in monkey striatum related to the expectation of predictable environmental events. , 1992, Journal of neurophysiology.

[9]  A. Lavoie,et al.  Spatial, movement- and reward-sensitive discharge by medial ventral striatum neurons of rats , 1994, Brain Research.

[10]  O. Hikosaka Role of Basal Ganglia in Control of Innate Movements, Learned Behavior and Cognition—A Hypothesis , 1994 .

[11]  J. Mink THE BASAL GANGLIA: FOCUSED SELECTION AND INHIBITION OF COMPETING MOTOR PROGRAMS , 1996, Progress in Neurobiology.

[12]  Masataka Watanabe Reward expectancy in primate prefrental neurons , 1996, Nature.

[13]  J. Hollerman,et al.  Dopamine neurons report an error in the temporal prediction of reward during learning , 1998, Nature Neuroscience.

[14]  O. Hikosaka,et al.  Expectation of reward modulates cognitive signals in the basal ganglia , 1998, Nature Neuroscience.

[15]  M. Shadlen,et al.  Effect of Expected Reward Magnitude on the Response of Neurons in the Dorsolateral Prefrontal Cortex of the Macaque , 1999, Neuron.

[16]  P. Redgrave,et al.  The basal ganglia: a vertebrate solution to the selection problem? , 1999, Neuroscience.

[17]  M. Jung,et al.  Fast spiking and regular spiking neural correlates of fear conditioning in the medial prefrontal cortex of the rat. , 2001, Cerebral cortex.

[18]  B. Knowlton,et al.  Learning and memory functions of the Basal Ganglia. , 2002, Annual review of neuroscience.

[19]  B. Everitt,et al.  Emotion and motivation: the role of the amygdala, ventral striatum, and prefrontal cortex , 2002, Neuroscience & Biobehavioral Reviews.

[20]  J. Salamone,et al.  Motivational views of reinforcement: implications for understanding the behavioral functions of nucleus accumbens dopamine , 2002, Behavioural Brain Research.

[21]  M. Jung,et al.  Dynamics of Population Code for Working Memory in the Prefrontal Cortex , 2003, Neuron.

[22]  G. Schoenbaum,et al.  Neural Encoding in Ventral Striatum during Olfactory Discrimination Learning , 2003, Neuron.

[23]  Howard L Fields,et al.  Cue-evoked firing of nucleus accumbens neurons encodes motivational significance during a discriminative stimulus task. , 2004, Journal of neurophysiology.

[24]  A. Redish,et al.  Neuronal activity in the rodent dorsal striatum in sequential navigation: separation of spatial and reward responses on the multiple T task. , 2004, Journal of neurophysiology.

[25]  M. Gluck,et al.  Cortico-striatal contributions to feedback-based learning: converging data from neuroimaging and neuropsychology. , 2004, Brain : a journal of neurology.

[26]  D. Barraclough,et al.  Prefrontal cortex and decision making in a mixed-strategy game , 2004, Nature Neuroscience.

[27]  T. Robbins,et al.  Putting a spin on the dorsal–ventral divide of the striatum , 2004, Trends in Neurosciences.

[28]  J. O'Doherty,et al.  Reward representations and reward-related learning in the human brain: insights from neuroimaging , 2004, Current Opinion in Neurobiology.

[29]  D. Barraclough,et al.  Reinforcement learning and decision making in monkeys during a competitive game. , 2004, Brain research. Cognitive brain research.

[30]  E. Vaadia,et al.  Coincident but Distinct Messages of Midbrain Dopamine and Striatal Tonically Active Neurons , 2004, Neuron.

[31]  K. Doya,et al.  Representation of Action-Specific Reward Values in the Striatum , 2005, Science.

[32]  E. Miller,et al.  Different time courses of learning-related activity in the prefrontal cortex and striatum , 2005, Nature.

[33]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[34]  P. Glimcher,et al.  Midbrain Dopamine Neurons Encode a Quantitative Reward Prediction Error Signal , 2005, Neuron.

[35]  P. Glimcher,et al.  JOURNAL OF THE EXPERIMENTAL ANALYSIS OF BEHAVIOR 2005, 84, 555–579 NUMBER 3(NOVEMBER) DYNAMIC RESPONSE-BY-RESPONSE MODELS OF MATCHING BEHAVIOR IN RHESUS MONKEYS , 2022 .

[36]  M. Roitman,et al.  Nucleus Accumbens Neurons Are Innately Tuned for Rewarding and Aversive Taste Stimuli, Encode Their Predictors, and Are Linked to Motor Output , 2005, Neuron.

[37]  K. Berridge The debate over dopamine’s role in reward: the case for incentive salience , 2007, Psychopharmacology.

[38]  Kae Nakamura,et al.  Basal ganglia orient eyes to reward. , 2006, Journal of neurophysiology.

[39]  W. Schultz Behavioral theories and the neurophysiology of reward. , 2006, Annual review of psychology.

[40]  K. Doya,et al.  The computational neurobiology of learning and reward , 2006, Current Opinion in Neurobiology.

[41]  B. Balleine,et al.  The Role of the Dorsal Striatum in Reward and Decision-Making , 2007, The Journal of Neuroscience.

[42]  Jeong‐Wook Ghim,et al.  Learning-Induced Enduring Changes in Functional Connectivity among Prefrontal Cortical Neurons , 2007, The Journal of Neuroscience.

[43]  Peter Redgrave,et al.  Basal Ganglia , 2020, Encyclopedia of Autism Spectrum Disorders.

[44]  R. O’Reilly,et al.  Separate neural substrates for skill learning and performance in the ventral and dorsal striatum , 2007, Nature Neuroscience.

[45]  M. Delgado,et al.  Reward‐Related Responses in the Human Striatum , 2007, Annals of the New York Academy of Sciences.

[46]  P. Glimcher,et al.  Action and Outcome Encoding in the Primate Caudate Nucleus , 2007, The Journal of Neuroscience.

[47]  H. Seo,et al.  Temporal Filtering of Reward Signals in the Dorsal Anterior Cingulate Cortex during a Mixed-Strategy Game , 2007, The Journal of Neuroscience.

[48]  E. Bézard,et al.  Shaping of Motor Responses by Incentive Values through the Basal Ganglia , 2007, The Journal of Neuroscience.

[49]  Daeyeol Lee,et al.  Encoding of action history in the rat ventral striatum. , 2007, Journal of neurophysiology.

[50]  M. Delgado,et al.  The role of the striatum in aversive learning and aversive prediction errors , 2008, Philosophical Transactions of the Royal Society B: Biological Sciences.

[51]  Steven P. Wise,et al.  Forward frontal fields: phylogeny and fundamental function , 2008, Trends in Neurosciences.

[52]  Simon Hong,et al.  New Insights on the Subcortical Representation of Reward This Review Comes from a Themed Issue on Cognitive Neuroscience Edited Lateral Habenula Serotonin Neurons , 2022 .

[53]  Simon Hong,et al.  The Globus Pallidus Sends Reward-Related Signals to the Lateral Habenula , 2008, Neuron.

[54]  P. Glimcher,et al.  Value Representations in the Primate Striatum during Matching Behavior , 2008, Neuron.

[55]  H. Seo,et al.  Cortical mechanisms for reinforcement learning in competitive games , 2008, Philosophical Transactions of the Royal Society B: Biological Sciences.

[56]  Daeyeol Lee,et al.  Behavioral and Neural Changes after Gains and Losses of Conditioned Reinforcers , 2009, The Journal of Neuroscience.

[57]  Daeyeol Lee,et al.  Valuation of uncertain and delayed rewards in primate prefrontal cortex , 2009, Neural Networks.

[58]  M. Laubach,et al.  Dynamic Encoding of Action Selection by the Medial Striatum , 2009, The Journal of Neuroscience.

[59]  E. Miller,et al.  Learning Substrates in the Primate Prefrontal Cortex and Striatum: Sustained Activity Related to Successful Actions , 2009, Neuron.

[60]  N. White Some highlights of research on the effects of caudate nucleus lesions over the past 200 years , 2009, Behavioural Brain Research.

[61]  K. Doya,et al.  Validation of Decision-Making Models and Analysis of Decision Variables in the Rat Basal Ganglia , 2009, The Journal of Neuroscience.

[62]  Namjung Huh,et al.  Model-based reinforcement learning under concurrent schedules of reinforcement in rodents. , 2009, Learning & memory.

[63]  H. Seo,et al.  Lateral Intraparietal Cortex and Reinforcement Learning during a Mixed-Strategy Game , 2009, Journal of Neuroscience.

[64]  Daniel Durstewitz,et al.  Comparing the prefrontal cortex of rats and primates: Insights from electrophysiology , 2008, Neurotoxicity Research.

[65]  A. Cooper,et al.  Predictive Reward Signal of Dopamine Neurons , 2011 .