Modulators of decision making

Human and animal decisions are modulated by a variety of environmental and intrinsic contexts. Here I consider computational factors that can affect decision making and review anatomical structures and neurochemical systems that are related to contextual modulation of decision making. Expectation of a high reward can motivate a subject to go for an action despite a large cost, a decision that is influenced by dopamine in the anterior cingulate cortex. Uncertainty of action outcomes can promote risk taking and exploratory choices, in which norepinephrine and the orbitofrontal cortex appear to be involved. Predictable environments should facilitate consideration of longer-delayed rewards, which depends on serotonin in the dorsal striatum and dorsal prefrontal cortex. This article aims to sort out factors that affect the process of decision making from the viewpoint of reinforcement learning theory and to bridge between such computational needs and their neurophysiological substrates.

[1]  J. R.,et al.  Quantitative analysis , 1892, Nature.

[2]  E. Rowland.  Theory of Games and Economic Behavior , 1946, Nature.

[3]  R J HERRNSTEIN,et al.  Relative and absolute strength of response as a function of frequency of reinforcement. , 1961, Journal of the experimental analysis of behavior.

[4]  A. Tversky,et al.  Prospect theory: analysis of decision under risk , 1979 .

[5]  W M Baum,et al.  Optimization and the matching law as accounts of instrumental behavior. , 1981, Journal of the experimental analysis of behavior.

[6]  D. Cross,et al.  Subjective probability and delay. , 1991, Journal of the experimental analysis of behavior.

[7]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[8]  J. Wickens,et al.  Dopamine reverses the depression of rat corticostriatal synapses which normally follows high-frequency stimulation of cortex In vitro , 1996, Neuroscience.

[9]  P. Dayan,et al.  A framework for mesencephalic dopamine systems based on predictive Hebbian learning , 1996, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[10]  Masataka Watanabe.  Reward expectancy in primate prefrental neurons , 1996, Nature.

[11]  David I. Laibson,et al.  Golden Eggs and Hyperbolic Discounting , 1997 .

[12]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[13]  Andrew G. Barto,et al.  Reinforcement learning , 1998 .

[14]  O. Hikosaka,et al.  Expectation of reward modulates cognitive signals in the basal ganglia , 1998, Nature Neuroscience.

[15]  Michael L. Platt,et al.  Neural correlates of decision variables in parietal cortex , 1999, Nature.

[16]  S. Mobini,et al.  Theory and method in the quantitative analysis of ”impulsive choice” behaviour: implications for psychopharmacology , 1999, Psychopharmacology.

[17]  T. Robbins,et al.  Tryptophan depletion impairs stimulus-reward learning while methylphenidate disrupts attentional control in healthy young adults: implications for the monoaminergic basis of impulsive behaviour , 1999, Psychopharmacology.

[18]  T. Robbins,et al.  Dissociable Deficits in the Decision-Making Cognition of Chronic Amphetamine Abusers, Opiate Abusers, Patients with Focal Damage to Prefrontal Cortex, and Tryptophan-Depleted Normal Volunteers: Evidence for Monoaminergic Mechanisms , 1999, Neuropsychopharmacology.

[19]  J. Hollerman,et al.  Reward processing in primate orbitofrontal cortex and basal ganglia. , 2000, Cerebral cortex.

[20]  K. Doya.  Complementary roles of basal ganglia and cerebellum in learning and motor control , 2000, Current Opinion in Neurobiology.

[21]  T. Robbins,et al.  Impulsive Choice Induced in Rats by Lesions of the Nucleus Accumbens Core , 2001, Science.

[22]  Junichiro Yoshimoto,et al.  Control of exploitation-exploration meta-parameter in reinforcement learning , 2002, Neural Networks.

[23]  Sham M. Kakade,et al.  Opponent interactions between serotonin and dopamine , 2002, Neural Networks.

[24]  M. Walton,et al.  The Role of Rat Medial Frontal Cortex in Effort-Based Decision Making , 2002, The Journal of Neuroscience.

[25]  Kenji Doya,et al.  Metalearning and neuromodulation , 2002, Neural Networks.

[26]  J. Deakin,et al.  Effects of quinolinic acid-induced lesions of the orbital prefrontal cortex on inter-temporal choice: a quantitative analysis , 2002, Psychopharmacology.

[27]  J. Deakin,et al.  Effects of lesions of the orbitofrontal cortex on sensitivity to delayed and probabilistic reinforcement , 2002, Psychopharmacology.

[28]  John N. J. Reynolds,et al.  Dopamine-dependent plasticity of corticostriatal synapses , 2002, Neural Networks.

[29]  S. Kakade,et al.  Acquisition and extinction in autoshaping. , 2002, Psychological review.

[30]  Samuel M. McClure,et al.  Temporal Prediction Errors in a Passive Learning Task Activate Human Striatum , 2003, Neuron.

[31]  Karl J. Friston,et al.  Temporal Difference Models and Reward-Related Learning in the Human Brain , 2003, Neuron.

[32]  B. Balleine,et al.  The role of prelimbic cortex in instrumental conditioning , 2003, Behavioural Brain Research.

[33]  Keiji Tanaka,et al.  Neuronal Correlates of Goal-Based Motor Selection in the Prefrontal Cortex , 2003, Science.

[34]  R. Rogers,et al.  Effects of beta-adrenoceptor blockade on components of human decision-making , 2004, Psychopharmacology.

[35]  Tatsuo K Sato,et al.  Correlated Coding of Motivation and Outcome of Decision by Dopamine Neurons , 2003, The Journal of Neuroscience.

[36]  Saori C. Tanaka,et al.  Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops , 2004, Nature Neuroscience.

[37]  J. Deakin,et al.  Effects of orbital prefrontal cortex dopamine depletion on inter-temporal choice: a quantitative analysis , 2004, Psychopharmacology.

[38]  W. Newsome,et al.  Matching Behavior and the Representation of Value in the Parietal Cortex , 2004, Science.

[39]  Karl J. Friston,et al.  Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning , 2004, Science.

[40]  Samuel M. McClure,et al.  Separate Neural Systems Value Immediate and Delayed Monetary Rewards , 2004, Science.

[41]  T. Robbins,et al.  Cognitive Inflexibility After Prefrontal Serotonin Depletion , 2004, Science.

[42]  M. Walton,et al.  Differential involvement of serotonin and dopamine systems in cost-benefit decisions about delay or effort , 2005, Psychopharmacology.

[43]  Trevor W Robbins,et al.  Fractionating Impulsivity: Contrasting Effects of Central 5-HT Depletion on Different Measures of Impulsive Behavior , 2004, Neuropsychopharmacology.

[44]  Alexander Strashny.  Asymmetric loss utility: an analysis of decision under risk , 2004 .

[45]  Peter Dayan,et al.  Temporal difference models describe higher-order learning in humans , 2004, Nature.

[46]  T. Robbins,et al.  Contrasting Roles of Basolateral Amygdala and Orbitofrontal Cortex in Impulsive Choice , 2004, The Journal of Neuroscience.

[47]  C. M. Bradshaw,et al.  The effect of orbital prefrontal cortex lesions on performance on a progressive ratio schedule: implications for models of inter-temporal choice , 2005, Behavioural Brain Research.

[48]  Camelia M. Kuhnen,et al.  The Neural Basis of Financial Risk Taking , 2005, Neuron.

[49]  K. Doya,et al.  Representation of Action-Specific Reward Values in the Striatum , 2005, Science.

[50]  Jonathan D. Cohen,et al.  An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. , 2005, Annual review of neuroscience.

[51]  S. Sara,et al.  Network reset: a simplified overarching theory of locus coeruleus noradrenaline function , 2005, Trends in Neurosciences.

[52]  T. Ono,et al.  Auditory thalamus integrates visual inputs into behavioral gains , 2005, Nature Neuroscience.

[53]  Angela J. Yu,et al.  Uncertainty, Neuromodulation, and Attention , 2005, Neuron.

[54]  M. Kimura,et al.  Complementary Process to Response Bias in the Centromedian Nucleus of the Thalamus , 2005, Science.

[55]  P. Dayan,et al.  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control , 2005, Nature Neuroscience.

[56]  B. Balleine.  Neural bases of food-seeking: Affect, arousal and reward in corticostriatolimbic circuits , 2005, Physiology & Behavior.

[57]  J. Feldon,et al.  Double dissociation of the effects of selective nucleus accumbens core and shell lesions on impulsive‐choice behaviour and salience learning in rats , 2005, The European journal of neuroscience.

[58]  Rudolf N Cardinal,et al.  Effects of lesions of the nucleus accumbens core on choice between small certain rewards and large uncertain rewards in rats , 2005, BMC Neuroscience.

[59]  A. Hariri,et al.  Preference for Immediate over Delayed Rewards Is Associated with Magnitude of Ventral Striatal Activity , 2006, The Journal of Neuroscience.

[60]  T. Robbins,et al.  Double dissociation between serotonergic and dopaminergic modulation of medial prefrontal and orbitofrontal cortex during a test of impulsive choice. , 2006, Cerebral cortex.

[61]  Timothy E. J. Behrens,et al.  Optimal decision making and the anterior cingulate cortex , 2006, Nature Neuroscience.

[62]  Rudolf N. Cardinal,et al.  Neural systems implicated in delayed and probabilistic reinforcement , 2006, Neural Networks.

[63]  T. Robbins,et al.  Neurochemical Modulation of Response Inhibition and Probabilistic Learning in Humans , 2006, Science.

[64]  B. Balleine,et al.  Parallel incentive processing: an integrated view of amygdala function , 2006, Trends in Neurosciences.

[65]  P. Dayan,et al.  Cortical substrates for exploratory decisions in humans , 2006, Nature.

[66]  M. Walton,et al.  Separate neural pathways process different decision costs , 2006, Nature Neuroscience.

[67]  L. Vanderschuren,et al.  Critical Involvement of Dopaminergic Neurotransmission in Impulsive Decision Making , 2006, Biological Psychiatry.

[68]  Angela J. Yu,et al.  Phasic norepinephrine: A neural interrupt signal for unexpected events , 2006, Network.

[69]  Kenji Doya,et al.  Humans Can Adopt Optimal Discounting Strategy under Real-Time Constraints , 2006, PLoS Comput. Biol..

[70]  Kenji Doya,et al.  Brain mechanism of reward prediction under predictable and unpredictable environmental dynamics , 2006, Neural Networks.

[71]  M. Walton,et al.  A Role for the Macaque Anterior Cingulate Gyrus in Social Valuation , 2006, Science.

[72]  J. Gläscher,et al.  Dissociable Systems for Gain- and Loss-Related Value Predictions and Errors of Prediction in the Human Brain , 2006, The Journal of Neuroscience.

[73]  P. Dayan,et al.  Tonic dopamine: opportunity costs and the control of response vigor , 2007, Psychopharmacology.

[74]  T. Robbins,et al.  Cognitive inflexibility after prefrontal serotonin depletion is behaviorally and neurochemically specific. , 2006, Cerebral cortex.

[75]  A. Tversky,et al.  Prospect theory: an analysis of decision under risk — Source link , 2007 .

[76]  R. Joosten,et al.  Medial prefrontal serotonin in the rat is involved in goal-directed behaviour when affect guides decision making , 2007, Psychopharmacology.

[77]  B. Balleine,et al.  The Role of the Dorsal Striatum in Reward and Decision-Making , 2007, The Journal of Neuroscience.

[78]  J. O'Doherty,et al.  Reward Value Coding Distinct From Risk Attitude-Related Uncertainty Coding in Human Reward Systems , 2006, Journal of neurophysiology.

[79]  Samuel M. McClure,et al.  Time Discounting for Primary Rewards , 2007, The Journal of Neuroscience.

[80]  O. Hikosaka,et al.  Lateral habenula as a source of negative reward signals in dopamine neurons , 2007, Nature.

[81]  Joseph J. Paton,et al.  Expectation Modulates Neural Responses to Pleasant and Aversive Stimuli in Primate Amygdala , 2007, Neuron.

[82]  Timothy E. J. Behrens,et al.  Learning the value of information in an uncertain world , 2007, Nature Neuroscience.

[83]  K. Doya.  Reinforcement learning: Computational theory and biological mechanisms , 2007, HFSP journal.

[84]  E. Bézard,et al.  Shaping of Motor Responses by Incentive Values through the Basal Ganglia , 2007, The Journal of Neuroscience.

[85]  G. Berns,et al.  Intertemporal choice – toward an integrative framework , 2007, Trends in Cognitive Sciences.

[86]  Saori C. Tanaka,et al.  Serotonin Differentially Regulates Short- and Long-Term Prediction of Rewards in the Ventral and Dorsal Striatum , 2007, PloS one.

[87]  P. Dayan,et al.  Differential Encoding of Losses and Gains in the Human Striatum , 2007, The Journal of Neuroscience.

[88]  P. Glimcher,et al.  The neural correlates of subjective value during intertemporal choice , 2007, Nature Neuroscience.

[89]  T. Robbins,et al.  Similar Effects of the Selective Noradrenaline Reuptake Inhibitor Atomoxetine on Three Distinct Forms of Impulsivity in the Rat , 2008, Neuropsychopharmacology.

[90]  S. Floresco,et al.  Dopaminergic and Glutamatergic Regulation of Effort- and Delay-Based Decision Making , 2008, Neuropsychopharmacology.