Annals of the New York Academy of Sciences Efficient Coding and the Neural Representation of Value

To survive in a dynamic environment, an organism must be able to effectively learn, store, and recall the expected benefits and costs of potential actions. The nature of the valuation and decision processes is thus of fundamental interest to researchers at the intersection of psychology, neuroscience, and economics. Although normative theories of choice have outlined the theoretical structure of these valuations, recent experiments have begun to reveal how value is instantiated in the activity of neurons and neural circuits. Here, we review the various forms of value coding that have been observed in different brain systems and examine the implications of these value representations for both neural circuits and behavior. In particular, we focus on emerging evidence that value coding in a number of brain areas is context dependent, varying as a function of both the current choice set and previously experienced values. Similar contextual modulation occurs widely in the sensory system, and efficient coding principles derived in the sensory domain suggest a new framework for understanding the neural coding of value.

[1]  E. Rowland Theory of Games and Economic Behavior , 1946, Nature.

[2]  P. Samuelson,et al.  Foundations of Economic Analysis. , 1948 .

[3]  F. Attneave Some informational aspects of visual perception. , 1954, Psychological review.

[4]  R J HERRNSTEIN,et al.  Relative and absolute strength of response as a function of frequency of reinforcement. , 1961, Journal of the experimental analysis of behavior.

[5]  A. Tversky Elimination by aspects: A theory of choice. , 1972 .

[6]  R. Duncan Luce,et al.  Individual Choice Behavior: A Theoretical Analysis , 1979 .

[7]  S. Lea,et al.  Contemporary Animal Learning Theory, Anthony Dickinson. Cambridge University Press, Cambridge (1981), xii, +177 pp. £12.50 hardback, £3.95 paperback , 1981 .

[8]  S. Laughlin A Simple Coding Procedure Enhances a Neuron's Information Capacity , 1981, Zeitschrift fur Naturforschung. Section C, Biosciences.

[9]  A G Barto,et al.  Toward a modern theory of adaptive networks: expectation and prediction. , 1981, Psychological review.

[10]  Christopher P. Puto,et al.  Adding Asymmetrically Dominated Alternatives: Violations of Regularity & the Similarity Hypothesis. , 1981 .

[11]  I. Ohzawa,et al.  Contrast gain control in the cat visual cortex , 1982, Nature.

[12]  J. Movshon,et al.  The statistical reliability of signals in single neurons in cat and monkey visual cortex , 1983, Vision Research.

[13]  C. Enroth-Cugell,et al.  Chapter 9 Visual adaptation and retinal gain controls , 1984 .

[14]  T. Wiesel,et al.  Intrinsic connectivity and receptive field properties in visual cortex , 1985, Vision Research.

[15]  W. Schultz,et al.  Dopamine neurons of the monkey midbrain: contingencies of responses to active touch during self-initiated arm movements. , 1990, Journal of neurophysiology.

[16]  W. Schultz,et al.  Responses of monkey dopamine neurons during learning of behavioral reactions. , 1992, Journal of neurophysiology.

[17]  D. Heeger Normalization of cell responses in cat striate cortex , 1992, Visual Neuroscience.

[18]  A. Tversky,et al.  Choice under Conflict: The Dynamics of Deferred Decision , 1992 .

[19]  William R. Softky,et al.  The highly irregular firing of cortical cells is inconsistent with temporal integration of random EPSPs , 1993, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[20]  I. Ohzawa,et al.  Length and width tuning of neurons in the cat's primary visual cortex. , 1994, Journal of neurophysiology.

[21]  W. Schultz,et al.  Importance of unpredictability for reward responses in primate dopamine neurons. , 1994, Journal of neurophysiology.

[22]  Peter Dayan,et al.  Bee foraging in uncertain environments using predictive hebbian learning , 1995, Nature.

[23]  P. Dayan,et al.  A framework for mesencephalic dopamine systems based on predictive Hebbian learning , 1996, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[24]  R. Wise,et al.  Addictive drugs and brain stimulation reward. , 1996, Annual review of neuroscience.

[25]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[26]  Michael J. Berry,et al.  Adaptation of retinal processing to image contrast and spatial scale , 1997, Nature.

[27]  David J. Field,et al.  Sparse coding with an overcomplete basis set: A strategy employed by V1? , 1997, Vision Research.

[28]  J. Movshon,et al.  Linearity and Normalization in Simple Cells of the Macaque Primary Visual Cortex , 1997, The Journal of Neuroscience.

[29]  J. Hollerman,et al.  Influence of reward expectation on behavior-related neuronal activity in primate striatum. , 1998, Journal of neurophysiology.

[30]  J. Hollerman,et al.  Dopamine neurons report an error in the temporal prediction of reward during learning , 1998, Nature Neuroscience.

[31]  M. Goldberg,et al.  The representation of visual salience in monkey parietal cortex , 1998, Nature.

[32]  W. Newsome,et al.  The Variable Discharge of Cortical Neurons: Implications for Connectivity, Computation, and Information Coding , 1998, The Journal of Neuroscience.

[33]  D. Ruderman,et al.  Independent component analysis of natural image sequences yields spatio-temporal filters similar to simple cells in primary visual cortex , 1998, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[34]  O. Hikosaka,et al.  Expectation of reward modulates cognitive signals in the basal ganglia , 1998, Nature Neuroscience.

[35]  D. Munoz,et al.  Saccadic Probability Influences Motor Preparation Signals and Time to Saccadic Initiation , 1998, The Journal of Neuroscience.

[36]  Michael L. Platt,et al.  Neural correlates of decision variables in parietal cortex , 1999, Nature.

[37]  W. Schultz,et al.  Relative reward preference in primate orbitofrontal cortex , 1999, Nature.

[38]  K. H. Britten,et al.  Spatial Summation in the Receptive Fields of MT Neurons , 1999, The Journal of Neuroscience.

[39]  William Bialek,et al.  Adaptive Rescaling Maximizes Information Transmission , 2000, Neuron.

[40]  M. Lepper,et al.  The Construction of Preference: When Choice Is Demotivating: Can One Desire Too Much of a Good Thing? , 2006 .

[41]  J L Gallant,et al.  Sparse coding and decorrelation in primary visual cortex during natural vision. , 2000, Science.

[42]  J. Price,et al.  The organization of networks within the orbital and medial prefrontal cortex of rats, monkeys and humans. , 2000, Cerebral cortex.

[43]  Adrienne L. Fairhall,et al.  Efficiency and ambiguity in an adaptive neural code , 2001, Nature.

[44]  Eero P. Simoncelli,et al.  Natural signal statistics and sensory gain control , 2001, Nature Neuroscience.

[45]  W. Schultz,et al.  Dopamine responses comply with basic assumptions of formal learning theory , 2001, Nature.

[46]  W. Newsome,et al.  Neural basis of a perceptual decision in the parietal cortex (area LIP) of the rhesus monkey. , 2001, Journal of neurophysiology.

[47]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[48]  Eero P. Simoncelli,et al.  Natural image statistics and neural representation. , 2001, Annual review of neuroscience.

[49]  S. Shafir,et al.  Context-dependent violations of rational choice in honeybees (Apis mellifera) and gray jays (Perisoreus canadensis) , 2001, Behavioral Ecology and Sociobiology.

[50]  K. H. Britten,et al.  Contrast dependence of response normalization in area MT of the rhesus macaque. , 2002, Journal of neurophysiology.

[51]  J. R. DeShazo,et al.  Designing Choice Sets for Stated Preference Methods: The Effects of Complexity on Choice Consistency , 2002 .

[52]  J. Movshon,et al.  Nature and interaction of signals from the receptive field center and surround in macaque V1 neurons. , 2002, Journal of neurophysiology.

[53]  M. Shadlen,et al.  Response of Neurons in the Lateral Intraparietal Area during a Combined Visual Discrimination Reaction Time Task , 2002, The Journal of Neuroscience.

[54]  J. Gallant,et al.  Natural Stimulation of the Nonclassical Receptive Field Increases Information Transmission Efficiency in V1 , 2002, The Journal of Neuroscience.

[55]  J. Assad,et al.  Dynamic coding of behaviourally relevant stimuli in parietal cortex , 2002, Nature.

[56]  J. Movshon,et al.  Time Course and Time-Distance Relationships for Surround Suppression in Macaque V1 Neurons , 2003, The Journal of Neuroscience.

[57]  Eero P. Simoncelli Vision and the statistics of the visual environment , 2003, Current Opinion in Neurobiology.

[58]  P. Lennie,et al.  Local signals from beyond the receptive fields of striate cortical neurons. , 2003, Journal of neurophysiology.

[59]  E. Miller,et al.  Neuronal activity in primate dorsolateral and orbital prefrontal cortex during performance of a reward preference task , 2003, The European journal of neuroscience.

[60]  R. Navarro,et al.  Optimal coding through divisive normalization models of V1 neurons. , 2003 .

[61]  Wolfram Schultz,et al.  Effects of expectations for different reward magnitudes on neuronal activity in primate striatum. , 2003, Journal of neurophysiology.

[62]  R. Navarro,et al.  Optimal coding through divisive normalization models of V1 neurons , 2003, Network.

[63]  M. Shadlen,et al.  Representation of Time by Neurons in the Posterior Parietal Cortex of the Macaque , 2003, Neuron.

[64]  Okihide Hikosaka,et al.  Reward-Dependent Gain and Bias of Visual Responses in Primate Superior Colliculus , 2003, Neuron.

[65]  T. A. Hurly,et al.  Context–dependent foraging decisions in rufous hummingbirds , 2003, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[66]  M. Goldberg,et al.  Neuronal Activity in the Lateral Intraparietal Area and Spatial Attention , 2003, Science.

[67]  R. Wise Dopamine, learning and motivation , 2004, Nature Reviews Neuroscience.

[68]  P. Glimcher,et al.  Activity in Posterior Parietal Cortex Is Correlated with the Relative Subjective Desirability of Action , 2004, Neuron.

[69]  W. Newsome,et al.  Matching Behavior and the Representation of Value in the Parietal Cortex , 2004, Science.

[70]  R. Andersen,et al.  Cognitive Control Signals for Neural Prosthetics , 2004, Science.

[71]  H. Ozeki,et al.  Relationship between Excitation and Inhibition Underlying Size Tuning and Contextual Response Modulation in the Cat Primary Visual Cortex , 2004, The Journal of Neuroscience.

[72]  S. Thorpe,et al.  The orbitofrontal cortex: Neuronal activity in the behaving monkey , 2004, Experimental Brain Research.

[73]  M. Roesch,et al.  Neuronal Activity Related to Reward Value and Motivation in Primate Frontal Cortex , 2004, Science.

[74]  J. Maunsell Neuronal representations of cognitive state: reward or attention? , 2004, Trends in Cognitive Sciences.

[75]  K. Doya,et al.  Representation of Action-Specific Reward Values in the Striatum , 2005, Science.

[76]  W. Schultz,et al.  Adaptive Coding of Reward Value by Dopamine Neurons , 2005, Science.

[77]  James J DiCarlo,et al.  Multiple Object Response Normalization in Monkey Inferotemporal Cortex , 2005, The Journal of Neuroscience.

[78]  J. Touryan,et al.  Contextual modulation of orientation tuning contributes to efficient processing of natural stimuli , 2005, Network.

[79]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[80]  P. Glimcher,et al.  Midbrain Dopamine Neurons Encode a Quantitative Reward Prediction Error Signal , 2005, Neuron.

[81]  C. Padoa-Schioppa,et al.  Neurons in the orbitofrontal cortex encode economic value , 2006, Nature.

[82]  Katherine I. Nagel,et al.  Temporal Processing and Adaptation in the Songbird Auditory Forebrain , 2006, Neuron.

[83]  Kenneth D. Miller,et al.  Adaptive filtering enhances information transmission in visual cortex , 2006, Nature.

[84]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[85]  Joseph J. Paton,et al.  The primate amygdala represents the positive and negative value of visual stimuli during learning , 2006, Nature.

[86]  O. Hikosaka,et al.  Comparison of Reward Modulation in the Frontal Eye Field and Caudate of the Macaque , 2006, The Journal of Neuroscience.

[87]  M. Carandini,et al.  The Statistical Computation Underlying Contrast Gain Control , 2006, The Journal of Neuroscience.

[88]  P. Dayan,et al.  Space and time in visual context , 2007, Nature Reviews Neuroscience.

[89]  A. Kohn Visual adaptation: physiology, mechanisms, and functional benefits. , 2007, Journal of neurophysiology.

[90]  Michael N. Shadlen,et al.  Probabilistic reasoning by neurons , 2007, Nature.

[91]  A. Fairhall,et al.  Shifts in Coding Properties and Maintenance of Information Transmission during Adaptation in Barrel Cortex , 2007, PLoS biology.

[92]  P. Glimcher,et al.  The neural correlates of subjective value during intertemporal choice , 2007, Nature Neuroscience.

[93]  Tai Sing Lee,et al.  Contextual Influences in Visual Processing , 2008 .

[94]  C. Padoa-Schioppa,et al.  The representation of economic value in the orbitofrontal cortex is invariant for changes of menu , 2008, Nature Neuroscience.

[95]  P. Glimcher,et al.  Value Representations in the Primate Striatum during Matching Behavior , 2008, Neuron.

[96]  Joseph J. Paton,et al.  Moment-to-Moment Tracking of State Value in the Amygdala , 2008, The Journal of Neuroscience.

[97]  M. Kimura,et al.  Neuronal encoding of reward value and direction of actions in the primate putamen. , 2009, Journal of neurophysiology.

[98]  D. Heeger,et al.  The Normalization Model of Attention , 2009, Neuron.

[99]  C. Padoa-Schioppa Range-Adapting Representation of Economic Value in the Orbitofrontal Cortex , 2009, The Journal of Neuroscience.

[100]  P. Glimcher,et al.  The Neurobiology of Decision: Consensus and Controversy , 2009, Neuron.

[101]  H. Seo,et al.  Lateral Intraparietal Cortex and Reinforcement Learning during a Mixed-Strategy Game , 2009, Journal of Neuroscience.

[102]  W. Schultz,et al.  Adaptation of Reward Sensitivity in Orbitofrontal Neurons , 2010, The Journal of Neuroscience.

[103]  Antonio Rangel,et al.  Neural computations associated with goal-directed choice , 2010, Current Opinion in Neurobiology.

[104]  M. Goldberg,et al.  Attention, intention, and priority in the parietal lobe. , 2010, Annual review of neuroscience.

[105]  James L. McClelland,et al.  Integration of Sensory and Reward Information during Perceptual Decision-Making in Lateral Intraparietal Cortex (LIP) of the Macaque Monkey , 2010, PloS one.

[106]  P. Glimcher Foundations of Neuroeconomic Analysis , 2010 .

[107]  V. Stuphorn,et al.  Supplementary eye field encodes option and action value for saccades with variable reward. , 2010, Journal of neurophysiology.

[108]  Kenway Louie,et al.  Separating Value from Choice: Delay Discounting Activity in the Lateral Intraparietal Area , 2010, The Journal of Neuroscience.

[109]  C. Padoa-Schioppa Neurobiology of economic choice: a good-based model. , 2011, Annual review of neuroscience.

[110]  P. Glimcher,et al.  Reward Value-Based Gain Control: Divisive Normalization in Parietal Cortex , 2011, The Journal of Neuroscience.

[111]  Paul Cisek,et al.  Neural Correlates of Biased Competition in Premotor Cortex , 2011, The Journal of Neuroscience.

[112]  P. Glimcher Understanding dopamine and reinforcement learning: The dopamine reward prediction error hypothesis , 2011, Proceedings of the National Academy of Sciences.

[113]  Joseph J Atick,et al.  Could information theory provide an ecological theory of sensory processing? , 2011, Network.

[114]  M. Carandini,et al.  Normalization as a canonical neural computation , 2011, Nature Reviews Neuroscience.

[115]  H. B. Barlow,et al.  Possible Principles Underlying the Transformations of Sensory Messages , 2012 .