Neural computations underlying action-based decision making in the human brain

Action-based decision making involves choices between different physical actions to obtain rewards. To make such decisions the brain needs to assign a value to each action and then compare them to make a choice. Using fMRI in human subjects, we found evidence for action-value signals in supplementary motor cortex. Separate brain regions, most prominently ventromedial prefrontal cortex, were involved in encoding the expected value of the action that was ultimately taken. These findings differentiate two main forms of value signals in the human brain: those relating to the value of each available action, likely reflecting signals that are a precursor of choice, and those corresponding to the expected value of the action that is subsequently chosen, and therefore reflecting the consequence of the decision process. Furthermore, we also found signals in the dorsomedial frontal cortex that resemble the output of a decision comparator, which implicates this region in the computation of the decision itself.

[1]  M. Roesch,et al.  Impact of expected reward on neuronal activity in prefrontal cortex, frontal and supplementary eye fields and premotor cortex. , 2003, Journal of neurophysiology.

[2]  R. Romo,et al.  Neuronal Correlates of a Perceptual Decision in Ventral Premotor Cortex , 2004, Neuron.

[3]  P. Glimcher,et al.  Midbrain Dopamine Neurons Encode a Quantitative Reward Prediction Error Signal , 2005, Neuron.

[4]  R. C. Oldfield The assessment and analysis of handedness: the Edinburgh inventory. , 1971, Neuropsychologia.

[5]  J. Gold,et al.  The Influence of Behavioral Context on the Representation of a Perceptual Decision in Developing Oculomotor Commands , 2003, The Journal of Neuroscience.

[6]  Kae Nakamura,et al.  Neuronal activity in medial frontal cortex during learning of sequential procedures. , 1998, Journal of neurophysiology.

[7]  S P Wise,et al.  The somatotopic organization of the supplementary motor area: intracortical microstimulation mapping , 1987, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[8]  Brian Knutson,et al.  Amygdalar Recruitment during Anticipation of Monetary Rewards , 2003, Annals of the New York Academy of Sciences.

[9]  P. Glimcher,et al.  Action and Outcome Encoding in the Primate Caudate Nucleus , 2007, The Journal of Neuroscience.

[10]  J. O'Doherty,et al.  Dissociating Valence of Outcome from Behavioral Control in Human Orbital and Ventral Prefrontal Cortices , 2003, The Journal of Neuroscience.

[11]  J Tanji,et al.  Changing directions of forthcoming arm movements: neuronal activity in the presupplementary and supplementary motor area of monkey cerebral cortex. , 1996, Journal of neurophysiology.

[12]  W. Newsome,et al.  A general mechanism for decision-making in the human brain? , 2005, Trends in Cognitive Sciences.

[13]  Brian Knutson,et al.  Dissociation of reward anticipation and outcome with event-related fMRI , 2001, Neuroreport.

[14]  H. Seo,et al.  Temporal Filtering of Reward Signals in the Dorsal Anterior Cingulate Cortex during a Mixed-Strategy Game , 2007, The Journal of Neuroscience.

[15]  Timothy E. J. Behrens,et al.  Optimal decision making and the anterior cingulate cortex , 2006, Nature Neuroscience.

[16]  O. Hikosaka,et al.  Neural Correlates of Rewarded and Unrewarded Eye Movements in the Primate Caudate Nucleus , 2003, The Journal of Neuroscience.

[17]  H. C Lau,et al.  Willed action and attention to the selection of action , 2004, NeuroImage.

[18]  M. Walton,et al.  Interactions between decision making and performance monitoring within prefrontal cortex , 2004, Nature Neuroscience.

[19]  M Tamai,et al.  Microstimulation of the supplementary eye field during saccade preparation , 1995, Neuroreport.

[20]  W. Newsome,et al.  Matching Behavior and the Representation of Value in the Parietal Cortex , 2004, Science.

[21]  Colin Camerer,et al.  Dissociating the Role of the Orbitofrontal Cortex and the Striatum in the Computation of Goal Values and Prediction Errors , 2008, The Journal of Neuroscience.

[22]  J. Neumann,et al.  Theory of games and economic behavior , 1945, 100 Years of Math Milestones.

[23]  J D Schall,et al.  Topography of supplementary eye field afferents to frontal eye field in macaque: Implications for mapping between saccade coordinate systems , 1993, Visual Neuroscience.

[24]  J. Hollerman,et al.  Modifications of reward expectation-related neuronal activity during learning in primate striatum. , 1998, Journal of neurophysiology.

[25]  B. Libet Unconscious cerebral initiative and the role of conscious will in voluntary action , 1985, Behavioral and Brain Sciences.

[26]  Okihide Hikosaka,et al.  Reward-Dependent Gain and Bias of Visual Responses in Primate Superior Colliculus , 2003, Neuron.

[27]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[28]  James L. McClelland,et al.  The time course of perceptual choice: the leaky, competing accumulator model. , 2001, Psychological review.

[29]  Karl J. Friston,et al.  Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning , 2004, Science.

[30]  Colin Camerer,et al.  A framework for studying the neurobiology of value-based decision making , 2008, Nature Reviews Neuroscience.

[31]  R. Romo,et al.  Temporal Evolution of a Decision-Making Process in Medial Premotor Cortex , 2002, Neuron.

[32]  R A Andersen,et al.  Supplementary motor area encodes reward expectancy in eye-movement tasks. , 2005, Journal of neurophysiology.

[33]  C. Padoa-Schioppa,et al.  Neurons in the orbitofrontal cortex encode economic value , 2006, Nature.

[34]  Leslie G. Ungerleider,et al.  The neural systems that mediate human perceptual decision making , 2008, Nature Reviews Neuroscience.

[35]  E. Rowland Theory of Games and Economic Behavior , 1946, Nature.

[36]  Michael L. Platt,et al.  Neural correlates of decision variables in parietal cortex , 1999, Nature.

[37]  Masataka Watanabe Reward expectancy in primate prefrental neurons , 1996, Nature.

[38]  J. Tanji,et al.  Neuronal activity in the supplementary and presupplementary motor areas for temporal organization of multiple movements. , 2000, Journal of neurophysiology.

[39]  E. Rolls,et al.  Abstract reward and punishment representations in the human orbitofrontal cortex , 2001, Nature Neuroscience.

[40]  Keiji Tanaka,et al.  Neuronal Correlates of Goal-Based Motor Selection in the Prefrontal Cortex , 2003, Science.

[41]  W. Newsome,et al.  Neural basis of a perceptual decision in the parietal cortex (area LIP) of the rhesus monkey. , 2001, Journal of neurophysiology.

[42]  M. Roesch,et al.  Neuronal Activity Related to Reward Value and Motivation in Primate Frontal Cortex , 2004, Science.

[43]  B. Balleine,et al.  The role of the dorsomedial striatum in instrumental conditioning , 2005, The European journal of neuroscience.

[44]  W. Schultz,et al.  Reward-related neuronal activity during go-nogo task performance in primate orbitofrontal cortex. , 2000, Journal of neurophysiology.

[45]  O. Hikosaka,et al.  Influence of reward expectation on visuospatial processing in macaque lateral prefrontal cortex. , 2002, Journal of neurophysiology.

[46]  Jerome R. Busemeyer,et al.  Computational Models of Decision Making , 2003 .

[47]  M. Brass,et al.  To Do or Not to Do: The Neural Signature of Self-Control , 2007, The Journal of Neuroscience.

[48]  J. Schlag,et al.  Unit activity related to spontaneous saccades in frontal dorsomedial cortex of monkey , 2004, Experimental Brain Research.

[49]  P. Dayan,et al.  A framework for mesencephalic dopamine systems based on predictive Hebbian learning , 1996, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[50]  J. Tanji,et al.  Differential roles of neuronal activity in the supplementary and presupplementary motor areas: from information retrieval to motor planning and execution. , 2004, Journal of neurophysiology.

[51]  G. Rizzolatti,et al.  Corticocortical connections of area F3 (SMA‐proper) and area F6 (pre‐SMA) in the macaque monkey , 1993, The Journal of comparative neurology.

[52]  W. Schultz,et al.  A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task , 1999, Neuroscience.

[53]  J. Movshon,et al.  A computational analysis of the relationship between neuronal and behavioral responses to visual motion , 1996, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[54]  B. Balleine,et al.  Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning , 2004, The European journal of neuroscience.

[55]  W. Schultz,et al.  Influence of expectation of different rewards on behavior-related neuronal activity in the striatum. , 2001, Journal of neurophysiology.

[56]  K. A. Hadland,et al.  The anterior cingulate and reward-guided selection of actions. , 2003, Journal of neurophysiology.

[57]  Jun Tanji,et al.  Distribution of eye- and arm-movement-related neuronal activity in the SEF and in the SMA and Pre-SMA of monkeys. , 2002, Journal of neurophysiology.

[58]  S. Wise,et al.  Neuronal activity in the supplementary eye field during acquisition of conditional oculomotor associations. , 1995, Journal of neurophysiology.

[59]  P. Strick,et al.  Muscle representation in the macaque motor cortex: an anatomical perspective. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[60]  J. O'Doherty,et al.  Encoding Predictive Reward Value in Human Amygdala and Orbitofrontal Cortex , 2003, Science.

[61]  P. Dayan,et al.  Cortical substrates for exploratory decisions in humans , 2006, Nature.

[62]  R. Passingham,et al.  The functions of the medial premotor cortex , 2004, Experimental Brain Research.

[63]  D. Barraclough,et al.  Prefrontal cortex and decision making in a mixed-strategy game , 2004, Nature Neuroscience.

[64]  S. Lehéricy,et al.  Foot, hand, face and eye representation in the human striatum. , 2003, Cerebral cortex.

[65]  M. Roesch,et al.  Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards , 2007, Nature Neuroscience.

[66]  C. Kennard,et al.  Human Medial Frontal Cortex Mediates Unconscious Inhibition of Voluntary Action , 2007, Neuron.

[67]  J. O'Doherty,et al.  Orbitofrontal Cortex Encodes Willingness to Pay in Everyday Economic Transactions , 2007, The Journal of Neuroscience.

[68]  J. Tanji,et al.  A motor area rostral to the supplementary motor area (presupplementary motor area) in the monkey: neuronal activity during a learned motor task. , 1992, Journal of neurophysiology.

[69]  G. S. Russo,et al.  Effect of eye position within the orbit on electrically elicited saccadic eye movements: a comparison of the macaque monkey's frontal and supplementary eye fields. , 1993, Journal of neurophysiology.

[70]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[71]  K. Doya,et al.  Representation of Action-Specific Reward Values in the Striatum , 2005, Science.

[72]  E. J. Tehovnik,et al.  Compensatory saccades made to remembered targets following orbital displacement by electrically stimulating the dorsomedial frontal cortex or frontal eye fields of primates , 1996, Brain Research.

[73]  P. H. Schiller,et al.  Conditional task-related responses in monkey dorsomedial frontal cortex , 2004, Experimental Brain Research.

[74]  Leslie G. Ungerleider,et al.  Involvement of human left dorsolateral prefrontal cortex in perceptual decision making is independent of response modality , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[75]  M. Schlag-Rey,et al.  Evidence for a supplementary eye field. , 1987, Journal of neurophysiology.

[76]  W. Newsome,et al.  Choosing the greater of two goods: neural currencies for valuation and decision making , 2005, Nature Reviews Neuroscience.

[77]  N. P. Bichot,et al.  Perceptual and motor processing stages identified in the activity of macaque frontal eye field neurons during visual search. , 1996, Journal of neurophysiology.

[78]  Tatsuo K Sato,et al.  Correlated Coding of Motivation and Outcome of Decision by Dopamine Neurons , 2003, The Journal of Neuroscience.

[79]  M. Delgado,et al.  Modulation of Caudate Activity by Action Contingency , 2004, Neuron.

[80]  B. Richmond,et al.  Anterior Cingulate: Single Neuronal Signals Related to Degree of Reward Expectancy , 2002, Science.

[81]  R. Andersen,et al.  Posterior Parietal Cortex Encodes Autonomously Selected Motor Plans , 2007, Neuron.

[82]  J Schlag,et al.  Reward-predicting and reward-detecting neuronal activity in the primate supplementary eye field. , 2000, Journal of neurophysiology.

[83]  J. Schall,et al.  Performance monitoring by the supplementary eye ® eld , 2000 .

[84]  G. Rizzolatti,et al.  Multiple representations of body movements in mesial area 6 and the adjacent cingulate cortex: An intracortical microstimulation study in the macaque monkey , 1991, The Journal of comparative neurology.

[85]  M. Botvinick,et al.  Anterior cingulate cortex, error detection, and the online monitoring of performance. , 1998, Science.

[86]  J. Tanji,et al.  Role for cingulate motor area cells in voluntary movement selection based on reward. , 1998, Science.

[87]  Nikos K Logothetis,et al.  The ins and outs of fMRI signals , 2007, Nature Neuroscience.

[88]  Jonathan D. Cohen,et al.  Anterior Cingulate Conflict Monitoring and Adjustments in Control , 2004, Science.

[89]  A. Graybiel,et al.  Distributed but convergent ordering of corticostriatal projections: analysis of the frontal eye field and the supplementary eye field in the macaque monkey , 1992, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[90]  J. Maunsell Neuronal representations of cognitive state: reward or attention? , 2004, Trends in Cognitive Sciences.

[91]  Wolfram Schultz,et al.  Effects of expectations for different reward magnitudes on neuronal activity in primate striatum. , 2003, Journal of neurophysiology.

[92]  J. Gläscher,et al.  Determining a role for ventromedial prefrontal cortex in encoding action-based value signals during reward-related decision making. , 2009, Cerebral cortex.

[93]  Michael I. Jordan,et al.  A more biologically plausible learning rule than backpropagation applied to a network model of cortical area 7a. , 1991, Cerebral cortex.

[94]  Jeffrey D. Schall,et al.  Relationship of presaccadic activity in frontal eye field and supplementary eye field to saccade initiation in macaque: Poisson spike train analysis , 2004, Experimental Brain Research.

[95]  M. Preul The Human Brain: Surface, Blood Supply, and Three-Dimensional Sectional Anatomy , 2001 .

[96]  K. Doya,et al.  A Neural Correlate of Reward-Based Behavioral Learning in Caudate Nucleus: A Functional Magnetic Resonance Imaging Study of a Stochastic Decision Task , 2004, The Journal of Neuroscience.

[97]  K. Hikosaka,et al.  Delay activity of orbital and lateral prefrontal neurons of the monkey varying with different rewards. , 2000, Cerebral cortex.

[98]  Philip L. Smith,et al.  Psychology and neurobiology of simple decisions , 2004, Trends in Neurosciences.

[99]  J. O'Doherty,et al.  The Role of the Ventromedial Prefrontal Cortex in Abstract State-Based Inference during Decision Making in Humans , 2006, The Journal of Neuroscience.

[100]  E. J. Tehovnik,et al.  Reversible inactivation of macaque dorsomedial frontal cortex: effects on saccades and fixations , 1999, Experimental Brain Research.

[101]  H. Seo,et al.  Dynamic signals related to choices and outcomes in the dorsolateral prefrontal cortex. , 2007, Cerebral cortex.

[102]  W. Fries Cortical projections to the superior colliculus in the macaque monkey: A retrograde study using horseradish peroxidase , 1984, The Journal of comparative neurology.