Separate encoding of model-based and model-free valuations in the human brain

Behavioral studies have long shown that humans solve problems in two ways, one intuitive and fast (System 1, model-free), and the other reflective and slow (System 2, model-based). The neurobiological basis of dual process problem solving remains unknown due to challenges of separating activation in concurrent systems. We present a novel neuroeconomic task that predicts distinct subjective valuation and updating signals corresponding to these two systems. We found two concurrent value signals in human prefrontal cortex: a System 1 model-free reinforcement signal and a System 2 model-based Bayesian signal. We also found a System 1 updating signal in striatal areas and a System 2 updating signal in lateral prefrontal cortex. Further, signals in prefrontal cortex preceded choices that are optimal according to either updating principle, while signals in anterior cingulate cortex and globus pallidus preceded deviations from optimal choice for reinforcement learning. These deviations tended to occur when uncertainty regarding optimal values was highest, suggesting that disagreement between dual systems is mediated by uncertainty rather than conflict, confirming recent theoretical proposals.

[1]  Karl J. Friston,et al.  Effective connectivity: Influence, causality and biophysical modeling , 2011, NeuroImage.

[2]  W. Schultz,et al.  Relative reward preference in primate orbitofrontal cortex , 1999, Nature.

[3]  Karl J. Friston,et al.  Temporal Difference Models and Reward-Related Learning in the Human Brain , 2003, Neuron.

[4]  J. O'Doherty,et al.  Orbitofrontal Cortex Encodes Willingness to Pay in Everyday Economic Transactions , 2007, The Journal of Neuroscience.

[5]  Wim De Neys,et al.  Conflict monitoring in dual process theories of thinking , 2008, Cognition.

[6]  Daniel N. Osherson,et al.  Functional neuroanatomy of deductive inference: A language-independent distributed network , 2007, NeuroImage.

[7]  D H Brainard,et al.  The Psychophysics Toolbox. , 1997, Spatial vision.

[8]  Masaki Tanaka,et al.  Enhanced modulation of neuronal activity during antisaccades in the primate globus pallidus. , 2009, Cerebral cortex.

[9]  P. Dayan,et al.  Cortical substrates for exploratory decisions in humans , 2006, Nature.

[10]  D. Kahneman,et al.  Heuristics and Biases: The Psychology of Intuitive Judgment , 2002 .

[11]  Jonathan D. Cohen,et al.  Anterior Cingulate Conflict Monitoring and Adjustments in Control , 2004, Science.

[12]  R. Rescorla.  A theory of pavlovian conditioning: The effectiveness of reinforcement and non-reinforcement , 1972 .

[13]  Jonathan Evans.  Dual-processing accounts of reasoning, judgment, and social cognition. , 2008, Annual review of psychology.

[14]  Makoto Ito,et al.  Evidence for Model-Based Action Planning in a Sequential Finger Movement Task , 2010, Journal of motor behavior.

[15]  C. Padoa-Schioppa,et al.  Neurons in the orbitofrontal cortex encode economic value , 2006, Nature.

[16]  P. Dayan,et al.  States versus Rewards: Dissociable Neural Prediction Error Signals Underlying Model-Based and Model-Free Reinforcement Learning , 2010, Neuron.

[17]  D. Kahneman,et al.  Frames and brains: elicitation and control of response tendencies , 2007, Trends in Cognitive Sciences.

[18]  J. O'Doherty,et al.  Regret and its avoidance: a neuroimaging study of choice behavior , 2005, Nature Neuroscience.

[19]  S. Haber,et al.  The Reward Circuit: Linking Primate Anatomy and Human Imaging , 2010, Neuropsychopharmacology.

[20]  Samuel M. McClure,et al.  Temporal Prediction Errors in a Passive Learning Task Activate Human Striatum , 2003, Neuron.

[21]  P. Bossaerts,et al.  The Impact of Disappointment in Decision Making: Inter-Individual Differences and Electrical Neuroimaging , 2011, Front. Hum. Neurosci..

[22]  S. Sloman.  The empirical case for two systems of reasoning. , 1996 .

[23]  Michael L. Platt,et al.  Neural correlates of decision variables in parietal cortex , 1999, Nature.

[24]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[25]  Colin Camerer,et al.  Neuroeconomics: decision making and the brain , 2008 .

[26]  P. Glimcher,et al.  The neural correlates of subjective value during intertemporal choice , 2007, Nature Neuroscience.

[27]  P. Glimcher,et al.  Title: the Neural Representation of Subjective Value under Risk and Ambiguity 1 2 , 2009 .

[28]  J. O'Doherty,et al.  Reward Value Coding Distinct From Risk Attitude-Related Uncertainty Coding in Human Reward Systems , 2006, Journal of neurophysiology.

[29]  P. Dayan,et al.  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control , 2005, Nature Neuroscience.

[30]  J. O'Doherty,et al.  The Role of the Ventromedial Prefrontal Cortex in Abstract State-Based Inference during Decision Making in Humans , 2006, The Journal of Neuroscience.

[31]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[32]  D. Kumaran,et al.  Frames, Biases, and Rational Decision-Making in the Human Brain , 2006, Science.

[33]  W. F. Prokasy,et al.  Classical conditioning II: Current research and theory. , 1972 .

[34]  D. Kahneman,et al.  Representativeness revisited: Attribute substitution in intuitive judgment. , 2002 .

[35]  Colin Camerer,et al.  Neural Systems Responding to Degrees of Uncertainty in Human Decision-Making , 2005, Science.

[36]  K. Preuschoff,et al.  The Neurobiological Foundations of Valuation in Human Decision Making Under Uncertainty , 2009 .

[37]  Samuel M. McClure,et al.  Separate Neural Systems Value Immediate and Delayed Monetary Rewards , 2004, Science.