Explicit neural signals reflecting reward uncertainty

The acknowledged importance of uncertainty in economic decision making has stimulated the search for neural signals that could influence learning and inform decision mechanisms. Current views distinguish two forms of uncertainty, namely risk and ambiguity, depending on whether the probability distributions of outcomes are known or unknown. Behavioural neurophysiological studies on dopamine neurons revealed a risk signal, which covaried with the standard deviation or variance of the magnitude of juice rewards and occurred separately from reward value coding. Human imaging studies identified similarly distinct risk signals for monetary rewards in the striatum and orbitofrontal cortex (OFC), thus fulfilling a requirement for the mean variance approach of economic decision theory. The orbitofrontal risk signal covaried with individual risk attitudes, possibly explaining individual differences in risk perception and risky decision making. Ambiguous gambles with incomplete probabilistic information induced stronger brain signals than risky gambles in OFC and amygdala, suggesting that the brain's reward system signals the partial lack of information. The brain can use the uncertainty signals to assess the uncertainty of rewards, influence learning, modulate the value of uncertain rewards and make appropriate behavioural choices between only partly known options.

[1]  J. Neumann,et al.  Theory of games and economic behavior , 1945, 100 Years of Math Milestones.

[2]  E. Rowland Theory of Games and Economic Behavior , 1946, Nature.

[3]  J. Tobin Liquidity Preference as Behavior towards Risk , 1958 .

[4]  Oskar Morgenstern The Theory of Games , 1960 .

[5]  D. Ellsberg Decision, probability, and utility: Risk, ambiguity, and the Savage axioms , 1961 .

[6]  R. Rescorla,et al.  A theory of Pavlovian conditioning : Variations in the effectiveness of reinforcement and nonreinforcement , 1972 .

[7]  N. Mackintosh A Theory of Attention: Variations in the Associability of Stimuli with Reinforcement , 1975 .

[8]  A. Tversky,et al.  Prospect theory: analysis of decision under risk , 1979 .

[9]  A Houston,et al.  The application of statistical decision theory to animal behaviour. , 1980, Journal of theoretical biology.

[10]  T. Caraco,et al.  An empirical demonstration of risk-sensitive foraging preferences , 1980, Animal Behaviour.

[11]  J. Pearce,et al.  A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. , 1980, Psychological review.

[12]  J. Pearce,et al.  A model for Pavlovian learning: Variations in the effectiveness of conditioned but not of unconditioned stimuli. , 1980 .

[13]  D. Harper Competitive foraging in mallards: “Ideal free’ ducks , 1982, Animal Behaviour.

[14]  D. Hubel,et al.  Segregation of form, color, movement, and depth: anatomy, physiology, and perception. , 1988, Science.

[15]  T. Caraco,et al.  Risk-sensitivity: ambient temperature affects foraging choice , 1990, Animal Behaviour.

[16]  Andrew B. Abel,et al.  Asset Prices Under Habit Formation and Catching Up with the Joneses , 1990 .

[17]  L A Real,et al.  Animal choice behavior and the evolution of cognitive architecture , 1991, Science.

[18]  A. Damasio,et al.  Insensitivity to future consequences following damage to human prefrontal cortex , 1994, Cognition.

[19]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[20]  Masataka Watanabe Reward expectancy in primate prefrental neurons , 1996, Nature.

[21]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[22]  Anna Nagurney,et al.  Foundations of Financial Economics , 1997 .

[23]  Michele A. Basso,et al.  Modulation of neuronal activity by target uncertainty , 1997, Nature.

[24]  E. Weber,et al.  Perceived risk attitudes: relating risk perception to risky choice , 1997 .

[25]  Andrew G. Barto,et al.  Reinforcement learning , 1998 .

[26]  Michael L. Platt,et al.  Neural correlates of decision variables in parietal cortex , 1999, Nature.

[27]  C. Plott,et al.  Basic Principles of Asset Pricing Theory: Evidence from Large-Scale Experimental Financial Markets , 2000 .

[28]  A. Damasio,et al.  Emotion, decision making and the orbitofrontal cortex. , 2000, Cerebral cortex.

[29]  L. Nystrom,et al.  Tracking the hemodynamic responses to reward and punishment in the striatum. , 2000, Journal of neurophysiology.

[30]  N. Logothetis,et al.  Neurophysiological investigation of the basis of the fMRI signal , 2001, Nature.

[31]  Charles A. Holt,et al.  Risk Aversion and Incentive Effects , 2002 .

[32]  J. Deakin,et al.  Effects of lesions of the orbitofrontal cortex on sensitivity to delayed and probabilistic reinforcement , 2002, Psychopharmacology.

[33]  B. Richmond,et al.  Anterior Cingulate: Single Neuronal Signals Related to Degree of Reward Expectancy , 2002, Science.

[34]  Brian Knutson,et al.  A region of mesial prefrontal cortex tracks monetarily rewarding outcomes: characterization with rapid event-related fMRI , 2003, NeuroImage.

[35]  Reid Hastie,et al.  Phineas gauged: decision-making and the human prefrontal cortex , 2003, Neuropsychologia.

[36]  R. Elliott,et al.  Differential Response Patterns in the Striatum and Orbitofrontal Cortex to Financial Reward in Humans: A Parametric Functional Magnetic Resonance Imaging Study , 2003, The Journal of Neuroscience.

[37]  Wolfram Schultz,et al.  Effects of expectations for different reward magnitudes on neuronal activity in primate striatum. , 2003, Journal of neurophysiology.

[38]  W. Schultz,et al.  Discrete Coding of Reward Probability and Uncertainty by Dopamine Neurons , 2003, Science.

[39]  J. C. Crowley,et al.  Saccade Reward Signals in Posterior Cingulate Cortex , 2003, Neuron.

[40]  James L. McClelland,et al.  A reexamination of the evidence for the somatic marker hypothesis: what participants really know in the Iowa gambling task. , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[41]  R. Andersen,et al.  Cognitive Control Signals for Neural Prosthetics , 2004, Science.

[42]  E. Weber,et al.  Predicting Risk-Sensitivity in Humans and Lower Animals: Risk as Variance or Coefficient of Variation , 2004, Psychological review.

[43]  Karen I. Bolla,et al.  Neural substrates of faulty decision-making in abstinent marijuana users , 2005, NeuroImage.

[44]  M. Platt,et al.  Risk-sensitive neurons in macaque posterior cingulate cortex , 2005, Nature Neuroscience.

[45]  Matthew T. Kaufman,et al.  Distributed Neural Representation of Expected Value , 2005, The Journal of Neuroscience.

[46]  K. Doya,et al.  Representation of Action-Specific Reward Values in the Striatum , 2005, Science.

[47]  Colin Camerer,et al.  Neural Systems Responding to Degrees of Uncertainty in Human Decision-Making , 2005, Science.

[48]  W. Schultz,et al.  Adaptive Coding of Reward Value by Dopamine Neurons , 2005, Science.

[49]  P. C. Fletcher,et al.  Abnormal frontal activations related to decision-making in current and former amphetamine and opiate dependent individuals , 2005, Psychopharmacology.

[50]  G. McCarthy,et al.  Decisions under Uncertainty: Probabilistic Context Influences Activation of Prefrontal and Parietal Cortices , 2005, The Journal of Neuroscience.

[51]  C. Padoa-Schioppa,et al.  Neurons in the orbitofrontal cortex encode economic value , 2006, Nature.

[52]  Joseph J. Paton,et al.  The primate amygdala represents the positive and negative value of visual stimuli during learning , 2006, Nature.

[53]  S. Quartz,et al.  Neural Differentiation of Expected Reward and Risk in Human Subcortical Structures , 2006, Neuron.

[54]  A. Lawrence,et al.  The somatic marker hypothesis: A critical evaluation , 2006, Neuroscience & Biobehavioral Reviews.

[55]  Evan M. Gordon,et al.  Neural Signatures of Economic Preferences for Risk and Ambiguity , 2006, Neuron.

[56]  A. Tversky,et al.  Prospect theory: an analysis of decision under risk — Source link , 2007 .

[57]  J. O'Doherty,et al.  Reward Value Coding Distinct From Risk Attitude-Related Uncertainty Coding in Human Reward Systems , 2006, Journal of neurophysiology.

[58]  K. Preuschoff,et al.  Adding Prediction Risk to the Theory of Reward Learning , 2007, Annals of the New York Academy of Sciences.

[59]  Richard S. Sutton,et al.  Reinforcement Learning , 1992, Handbook of Machine Learning.