Using a Simple Neural Network to Delineate Some Principles of Distributed Economic Choice

The brain uses a mixture of distributed and modular organization to perform computations and generate appropriate actions. While the principles under which the brain might perform computations using modular systems have been more amenable to modeling, the principles by which the brain might make choices using distributed principles have not been explored. Our goal in this perspective is to delineate some of those distributed principles using a neural network method and use its results as a lens through which to reconsider some previously published neurophysiological data. To allow for direct comparison with our own data, we trained the neural network to perform binary risky choices. We find that value correlates are ubiquitous and are always accompanied by non-value information, including spatial information (i.e., no pure value signals). Evaluation, comparison, and selection were not distinct processes; indeed, value signals even in the earliest stages contributed directly, albeit weakly, to action selection. There was no place, other than at the level of action selection, at which dimensions were fully integrated. No units were specialized for specific offers; rather, all units encoded the values of both offers in an anti-correlated format, thus contributing to comparison. Individual network layers corresponded to stages in a continuous rotation from input to output space rather than to functionally distinct modules. While our network is likely to not be a direct reflection of brain processes, we propose that these principles should serve as hypotheses to be tested and evaluated for future studies.

[1]  A. Rangel,et al.  Informatic parcellation of the network involved in the computation of subjective value. , 2014, Social cognitive and affective neuroscience.

[2]  Paul W. Glimcher,et al.  Absence of Spatial Tuning in the Orbitofrontal Cortex , 2014, PloS one.

[3]  Steven T Piantadosi,et al.  Robust mixture modeling reveals category-free selectivity in reward region neuronal ensembles , 2016, bioRxiv.

[4]  C. Padoa-Schioppa,et al.  The representation of economic value in the orbitofrontal cortex is invariant for changes of menu , 2008, Nature Neuroscience.

[5]  Xiao-Jing Wang,et al.  Synaptic computation underlying probabilistic inference , 2010, Nature Neuroscience.

[6]  Xiao-Jing Wang,et al.  The importance of mixed selectivity in complex cognitive tasks , 2013, Nature.

[7]  Brianna J. Sleezer,et al.  Neuronal responses support a role for orbitofrontal cortex in cognitive set reconfiguration , 2016, bioRxiv.

[8]  J. O'Doherty,et al.  Evidence for a Common Representation of Decision Values for Dissimilar Goods in Human Ventromedial Prefrontal Cortex , 2009, The Journal of Neuroscience.

[9]  Andreas Wilke,et al.  Hot-hand bias in rhesus monkeys. , 2014, Journal of experimental psychology. Animal learning and cognition.

[10]  Daniel C. McNamee,et al.  Category-dependent and category-independent goal-value codes in human ventromedial prefrontal cortex , 2013, Nature Neuroscience.

[11]  Colin Camerer,et al.  A framework for studying the neurobiology of value-based decision making , 2008, Nature Reviews Neuroscience.

[12]  Joseph W. Kable,et al.  The valuation system: A coordinate-based meta-analysis of BOLD fMRI experiments examining neural correlates of subjective value , 2013, NeuroImage.

[13]  D. Plaut Double dissociation without modularity: evidence from connectionist neuropsychology. , 1995, Journal of clinical and experimental neuropsychology.

[14]  J. Kalaska,et al.  Neural Correlates of Reaching Decisions in Dorsal Premotor Cortex: Specification of Multiple Direction Choices and Final Selection of Action , 2005, Neuron.

[15]  Timothy E. J. Behrens,et al.  Review Frontal Cortex and Reward-guided Learning and Decision-making Figure 1. Frontal Brain Regions in the Macaque Involved in Reward-guided Learning and Decision-making Finer Grained Anatomical Divisions with Frontal Cortical Systems for Reward-guided Behavior , 2022 .

[16]  Timothy E. J. Behrens,et al.  Capturing the temporal evolution of choice across prefrontal cortex , 2015, eLife.

[17]  E. Fetz Movement control: Are movement parameters recognizably coded in the activity of single neurons? , 1992 .

[18]  Andrew W. Moore,et al.  Generalization in Reinforcement Learning: Safely Approximating the Value Function , 1994, NIPS.

[19]  Paul Cisek,et al.  Cortical mechanisms of action selection: the affordance competition hypothesis , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[20]  Tim Kovacs,et al.  On optimal decision-making in brains and social insect colonies , 2009, Journal of The Royal Society Interface.

[21]  Lara Jehi,et al.  Neuropsychological outcome following frontal lobectomy for pharmacoresistant epilepsy in adults , 2017, Neurology.

[22]  Tommy C. Blanchard,et al.  Monkeys Are More Patient in a Foraging Task than in a Standard Intertemporal Choice Task , 2015, PloS one.

[23]  J. Harlow Recovery from the passage of an iron bar through the head , 1993 .

[24]  Habiba Azab,et al.  Neuronal selectivity for spatial positions of offers and choices in five reward regions. , 2016, Journal of neurophysiology.

[25]  Xiao-Jing Wang,et al.  Neural mechanism for stochastic behaviour during a competitive game , 2006, Neural Networks.

[26]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[27]  Tommy C. Blanchard,et al.  Ramping ensemble activity in dorsal anterior cingulate neurons during persistent commitment to a decision. , 2015, Journal of neurophysiology.

[28]  T. Seeley,et al.  Group decision making in swarms of honey bees , 1999, Behavioral Ecology and Sociobiology.

[29]  M. Woolrich,et al.  Mechanisms underlying cortical activity during value-guided choice , 2011, Nature Neuroscience.

[30]  Michael A. Arbib,et al.  The handbook of brain theory and neural networks , 1995, A Bradford book.

[31]  Matthew T. Kaufman,et al.  Neural population dynamics during reaching , 2012, Nature.

[32]  J. Rinzel,et al.  Noise-induced alternations in an attractor network model of perceptual bistability. , 2007, Journal of neurophysiology.

[33]  Benjamin Y. Hayden,et al.  Differential Contributions of Ventral and Dorsal Striatum to Early and Late Phases of Cognitive Set Reconfiguration , 2016, Journal of Cognitive Neuroscience.

[34]  Brianna J. Sleezer,et al.  Rule Encoding in Orbitofrontal Cortex and Striatum Guides Selection , 2016, The Journal of Neuroscience.

[35]  Xiao-Jing Wang,et al.  Probabilistic Decision Making by Slow Reverberation in Cortical Circuits , 2002, Neuron.

[36]  C. Padoa-Schioppa,et al.  Neurons in the orbitofrontal cortex encode economic value , 2006, Nature.

[37]  Guillaume Sescousse,et al.  A common currency for the computation of motivational values in the human striatum. , 2015, Social cognitive and affective neuroscience.

[38]  B. Hayden,et al.  Correlates of decisional dynamics in the dorsal anterior cingulate cortex , 2017, PLoS biology.

[39]  S. Floresco,et al.  Dissociable Roles for the Nucleus Accumbens Core and Shell in Regulating Set Shifting , 2006, The Journal of Neuroscience.

[40]  M. Farah Neuropsychological inference with an interactive brain: A critique of the "locality" assumption. , 1994 .

[41]  M. Roesch,et al.  Encoding of Time-Discounted Rewards in Orbitofrontal Cortex Is Independent of Value Representation , 2006, Neuron.

[42]  Maria V. Sanchez-Vives,et al.  Lateral orbitofrontal cortex anticipates choices and integrates prior with current information , 2017, Nature Communications.

[43]  Tianming Yang,et al.  Covert shift of attention modulates the value encoding in the orbitofrontal cortex , 2017, bioRxiv.

[44]  Rachel Bitecofer,et al.  The Party Decides , 2018 .

[45]  C. Padoa-Schioppa,et al.  Neuronal Encoding of Subjective Value in Dorsal and Ventral Anterior Cingulate Cortex , 2012, The Journal of Neuroscience.

[46]  Daeyeol Lee,et al.  Ubiquity and Specificity of Reinforcement Signals throughout the Human Brain , 2011, Neuron.

[47]  C. Padoa-Schioppa Neurobiology of economic choice: a good-based model. , 2011, Annual review of neuroscience.

[48]  Habiba Azab,et al.  Shared roles of dorsal and subgenual anterior cingulate cortices in economic decisions , 2016, bioRxiv.

[49]  H. Damasio,et al.  Characterization of the decision-making deficit of patients with ventromedial prefrontal cortex lesions. , 2000, Brain : a journal of neurology.

[50]  Konrad Paul Kording,et al.  Could a Neuroscientist Understand a Microprocessor? , 2016, bioRxiv.

[51]  Aldo Genovesio,et al.  Monkey Orbitofrontal Cortex Encodes Response Choices Near Feedback Time , 2009, The Journal of Neuroscience.

[52]  James L. McClelland,et al.  Loss aversion and inhibition in dynamical models of multialternative choice. , 2004, Psychological review.

[53]  S. Kennerley,et al.  Evaluating choices by single neurons in the frontal lobe: outcome value encoded across multiple decision variables , 2009, The European journal of neuroscience.

[54]  J. O'Doherty,et al.  The problem with value , 2014, Neuroscience & Biobehavioral Reviews.

[55]  Erin L. Rich,et al.  Decoding subjective decisions from orbitofrontal cortex , 2016, Nature Neuroscience.

[56]  R. Dolan,et al.  Temporally Dissociable Contributions of Human Medial Prefrontal Subregions to Reward-Guided Learning , 2015, The Journal of Neuroscience.

[57]  Colin Camerer,et al.  Self-control in decision-making involves modulation of the vmPFC valuation system , 2009, NeuroImage.

[58]  W. Newsome,et al.  Orbitofrontal Cortex Value Signals Depend on Fixation Location during Free Viewing , 2016, Neuron.

[59]  J. Movshon,et al.  A computational analysis of the relationship between neuronal and behavioral responses to visual motion , 1996, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[60]  M. Quirk,et al.  Representation of Spatial Goals in Rat Orbitofrontal Cortex , 2006, Neuron.

[61]  Xiao-Jing Wang,et al.  A Recurrent Network Mechanism of Time Integration in Perceptual Decisions , 2006, The Journal of Neuroscience.

[62]  A. Cravo,et al.  The relation between action, predictability and temporal contiguity in temporal binding. , 2011, Acta psychologica.

[63]  Brianna J. Sleezer,et al.  Signatures of Value Comparison in Ventral Striatum Neurons , 2015, PLoS biology.

[64]  Benjamin Y Hayden,et al.  Preference patterns for skewed gambles in rhesus monkeys , 2013, Biology Letters.

[65]  Benjamin Y. Hayden,et al.  Time discounting and time preference in animals: A critical review , 2016, Psychonomic bulletin & review.

[66]  Tommy C. Blanchard,et al.  Reward Value Comparison via Mutual Inhibition in Ventromedial Prefrontal Cortex , 2014, Neuron.

[67]  M. Vinck,et al.  Theta-Band Phase Locking of Orbitofrontal Neurons during Reward Expectancy , 2010, The Journal of Neuroscience.

[68]  F. Ratnieks Honeybee Democracy Thomas D. Seeley Honeybee Democracy , 2011, Animal Behaviour.

[69]  E. Miller,et al.  An integrative theory of prefrontal cortex function. , 2001, Annual review of neuroscience.

[70]  Larry M. Bartels Presidential Primaries and the Dynamics of Public Choice , 1988 .

[71]  Benjamin Y. Hayden,et al.  Control without controllers: Towards a distributed neuroscience of executive control , 2016, bioRxiv.

[72]  Mark F Bear,et al.  Reward timing in the primary visual cortex. , 2006, Science.

[73]  J. O'Doherty,et al.  Reward representations and reward-related learning in the human brain: insights from neuroimaging , 2004, Current Opinion in Neurobiology.

[74]  J. Kalaska,et al.  Neural mechanisms for interacting with a world full of action choices. , 2010, Annual review of neuroscience.

[75]  M. Roesch,et al.  Executive Control Signals in Orbitofrontal Cortex during Response Inhibition , 2015, The Journal of Neuroscience.

[76]  Timothy E. J. Behrens,et al.  Double dissociation of value computations in orbitofrontal and anterior cingulate neurons , 2011, Nature Neuroscience.

[77]  Mark G. Baxter,et al.  Functional localization within the prefrontal cortex: missing the forest for the trees? , 2010, Trends in Neurosciences.

[78]  Dino J. Levy,et al.  The root of all value: a neural common currency for choice , 2012, Current Opinion in Neurobiology.

[79]  H. Francis Song,et al.  Clustering and compositionality of task representations in a neural network trained to perform many cognitive tasks , 2017, bioRxiv.

[80]  K. C. Anderson,et al.  Single neurons in prefrontal cortex encode abstract rules , 2001, Nature.

[81]  K. Tsutsui,et al.  Rule-dependent anticipatory activity in prefrontal neurons , 2010, Neuroscience Research.

[82]  John M. Pearson,et al.  Decision Making: The Neuroethological Turn , 2014, Neuron.

[83]  J. Bickle,et al.  NEUROECONOMICS, NEUROPHYSIOLOGY AND THE COMMON CURRENCY HYPOTHESIS , 2008, Economics and Philosophy.

[84]  David M. Himmelblau,et al.  Process control via artificial neural networks and reinforcement learning , 1992 .

[85]  Wei Ji Ma,et al.  Efficient Probabilistic Inference in Generic Neural Networks Trained with Non-Probabilistic Feedback , 2018 .

[86]  Chung-Hay Luk,et al.  Choice Coding in Frontal Cortex during Stimulus-Guided or Action-Guided Decision-Making , 2013, The Journal of Neuroscience.

[87]  John T Serences,et al.  Value-Based Modulations in Human Visual Cortex , 2008, Neuron.

[88]  B. Hayden Economic choice: the foraging perspective , 2018, Current Opinion in Behavioral Sciences.

[89]  B. Hayden,et al.  Dorsal anterior cingulate: a Rorschach test for cognitive neuroscience , 2016, Nature Neuroscience.

[90]  M. Roesch,et al.  Rule encoding in dorsal striatum impacts action selection , 2015, The European journal of neuroscience.

[91]  Nils Kolling,et al.  A neural mechanism underlying failure of optimal choice with multiple alternatives , 2014, Nature Neuroscience.

[92]  P. Cisek Making decisions through a distributed consensus , 2012, Current Opinion in Neurobiology.

[93]  Colin Camerer,et al.  Dissociating the Role of the Orbitofrontal Cortex and the Striatum in the Computation of Goal Values and Prediction Errors , 2008, The Journal of Neuroscience.

[94]  Melanie Mitchell,et al.  Complexity - A Guided Tour , 2009 .

[95]  Timothy D. Hanks,et al.  Probabilistic Population Codes for Bayesian Decision Making , 2008, Neuron.

[96]  Benjamin Y Hayden,et al.  Dorsal Anterior Cingulate Cortex: A Bottom-Up View. , 2016, Annual review of neuroscience.

[97]  Jonathan D. Wallis,et al.  Medial-lateral Organization of the Orbitofrontal Cortex , 2014, Journal of Cognitive Neuroscience.

[98]  Benjamin Y. Hayden,et al.  Reactivation of associative structure specific outcome responses during prospective evaluation in reward-based choices , 2017, Nature Communications.

[99]  Marvin Minsky,et al.  Perceptrons: An Introduction to Computational Geometry , 1969 .

[100]  H. Seo,et al.  Temporal Filtering of Reward Signals in the Dorsal Anterior Cingulate Cortex during a Mixed-Strategy Game , 2007, The Journal of Neuroscience.

[101]  Tommy C. Blanchard,et al.  Orbitofrontal Cortex Uses Distinct Codes for Different Choice Attributes in Decisions Motivated by Curiosity , 2015, Neuron.

[102]  Tomoyuki Furuyashiki,et al.  Rat Orbitofrontal Cortex Separately Encodes Response and Outcome Information during Performance of Goal-Directed Behavior , 2008, The Journal of Neuroscience.

[103]  I. Couzin Collective cognition in animal groups , 2009, Trends in Cognitive Sciences.

[104]  Colin Camerer,et al.  Transformation of stimulus value signals into motor commands during simple choice , 2011, Proceedings of the National Academy of Sciences.

[105]  Jack L. Gallant,et al.  Working Memory and Decision Processes in Visual Area V4 , 2013, Front. Neurosci..

[106]  Jung Hoon Sul,et al.  Distinct Roles of Rodent Orbitofrontal and Medial Prefrontal Cortex in Decision Making , 2010, Neuron.

[107]  Eamonn B. Mallon,et al.  Information flow, opinion polling and collective intelligence in house-hunting social insects. , 2002, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[108]  E. Miller,et al.  From rule to response: neuronal processes in the premotor and prefrontal cortex. , 2003, Journal of neurophysiology.

[109]  Tianming Yang,et al.  Covert Shift of Attention Modulates the Value Encoding in the Orbitofrontal Cortex , 2017, bioRxiv.

[110]  Erin L. Rich,et al.  Challenges of Interpreting Frontal Neurons during Value-Based Decision-Making , 2011, Front. Neurosci..

[111]  Benjamin Y. Hayden,et al.  Human Neuroscience , 2022 .

[112]  B. Hayden,et al.  A distributed, hierarchical and recurrent framework for reward-based choice , 2017, Nature Reviews Neuroscience.

[113]  Daeyeol Lee,et al.  Distributed Coding of Actual and Hypothetical Outcomes in the Orbital and Dorsolateral Prefrontal Cortex , 2011, Neuron.

[114]  C. Padoa-Schioppa,et al.  Multi-stage mental process for economic choice in capuchins , 2006, Cognition.

[115]  C. Padoa-Schioppa,et al.  A neuro-computational model of economic decisions. , 2015, Journal of neurophysiology.