An integrate-and-fire model of prefrontal cortex neuronal activity during performance of goal-directed decision making.

The orbital frontal cortex appears to be involved in learning the rules of goal-directed behavior necessary to perform the correct actions based on perception to accomplish different tasks. The activity of orbitofrontal neurons changes dependent upon the specific task or goal involved, but the functional role of this activity in performance of specific tasks has not been fully determined. Here we present a model of prefrontal cortex function using networks of integrate-and-fire neurons arranged in minicolumns. This network model forms associations between representations of sensory input and motor actions, and uses these associations to guide goal-directed behavior. The selection of goal-directed actions involves convergence of the spread of activity from the goal representation with the spread of activity from the current state. This spiking network model provides a biological implementation of the action selection process used in reinforcement learning theory. The spiking activity shows properties similar to recordings of orbitofrontal neurons during task performance.

[1]  J. Konorski Conditioned reflexes and neuron organization. , 1948 .

[2]  R. Stein Some models of neuronal variability. , 1967, Biophysical journal.

[3]  D. J. White,et al.  Dynamic Programming , 2018, Wiley Encyclopedia of Computer Science and Engineering.

[4]  R. Rescorla A theory of pavlovian conditioning: The effectiveness of reinforcement and non-reinforcement , 1972 .

[5]  J. Fuster Unit activity in prefrontal cortex during delayed-response performance: neuronal correlates of transient memory. , 1973, Journal of neurophysiology.

[6]  T. Bliss,et al.  Long‐lasting potentiation of synaptic transmission in the dentate area of the anaesthetized rabbit following stimulation of the perforant path , 1973, The Journal of physiology.

[7]  R. Bartus,et al.  Short-term memory in the rhesus monkey: Disruption from the anti-cholinergic scopolamine , 1976, Pharmacology Biochemistry and Behavior.

[8]  A G Barto,et al.  Toward a modern theory of adaptive networks: expectation and prediction. , 1981, Psychological review.

[9]  J. Fuster,et al.  Cellular discharge in the dorsolateral prefrontal cortex of the monkey in cognitive tasks , 1982, Experimental Neurology.

[10]  W. Levy,et al.  Temporal contiguity requirements for long-term associative potentiation/depression in the hippocampus , 1983, Neuroscience.

[11]  D. Penetar,et al.  Effects of cholinergic drugs on delayed match-to-sample performance of rhesus monkeys , 1983, Pharmacology Biochemistry and Behavior.

[12]  Sheldon M. Ross Discounted Dynamic Programming , 1983 .

[13]  John H. R. Maunsell,et al.  The connections of the middle temporal visual area (MT) and their relationship to a cortical hierarchy in the macaque monkey , 1983, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[14]  David Harel,et al.  Statecharts: A Visual Formalism for Complex Systems , 1987, Sci. Comput. Program..

[15]  W Buño,et al.  Cross-correlation analysis of septohippocampal neurons during theta-rhythm. , 1987, Brain research.

[16]  A. Alonso,et al.  Cross-correlation analysis of septohippocampal neurons during ≡-rhythm , 1987, Brain Research.

[17]  Kevan A. C. Martin,et al.  A Canonical Microcircuit for Neocortex , 1989, Neural Computation.

[18]  P. Goldman-Rakic,et al.  Mnemonic coding of visual space in the monkey's dorsolateral prefrontal cortex. , 1989, Journal of neurophysiology.

[19]  D. Pandya,et al.  Architecture and intrinsic connections of the prefrontal cortex in the rhesus monkey , 1989, The Journal of comparative neurology.

[20]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[21]  W. Singer,et al.  Different voltage-dependent thresholds for inducing long-term depression and long-term potentiation in slices of rat visual cortex , 1990, Nature.

[22]  M. Stewart,et al.  Do septal neurons pace the hippocampal theta rhythm? , 1990, Trends in Neurosciences.

[23]  William A. Phillips,et al.  A Biologically Supported Error-Correcting Learning Rule , 1991, Neural Computation.

[24]  Rodrigo Andrade,et al.  Cell excitation enhances muscarinic cholinergic responses in rat association cortex , 1991, Brain Research.

[25]  D. J. Felleman,et al.  Distributed hierarchical processing in the primate cerebral cortex. , 1991, Cerebral cortex.

[26]  Jeffrey L. Elman,et al.  Distributed Representations, Simple Recurrent Networks, and Grammatical Structure , 1991, Mach. Learn..

[27]  J. Cohen,et al.  Context, cortex, and dopamine: a connectionist approach to behavior and biology in schizophrenia. , 1992, Psychological review.

[28]  Terrence J. Sejnowski,et al.  Using Aperiodic Reinforcement for Directed Self-Organization During Development , 1992, NIPS.

[29]  J. Fuster,et al.  Mnemonic and predictive functions of cortical neurons in a memory task , 1992, Neuroreport.

[30]  D Mumford,et al.  On the computational architecture of the neocortex. II. The role of cortico-cortical loops. , 1992, Biological cybernetics.

[31]  J. B. Levitt,et al.  Comparison of intrinsic connectivity in different areas of macaque monkey cerebral cortex. , 1993, Cerebral cortex.

[32]  T. Bliss,et al.  A synaptic model of memory: long-term potentiation in the hippocampus , 1993, Nature.

[33]  Mitsuo Kawato,et al.  A forward-inverse optics model of reciprocal connections between visual cortical areas , 1993 .

[34]  Joel L. Davis,et al.  Large-Scale Neuronal Theories of the Brain , 1994 .

[35]  T. Sejnowski,et al.  The predictive brain: temporal coincidence and temporal order in synaptic learning mechanisms. , 1994, Learning & memory.

[36]  A. Damasio,et al.  Insensitivity to future consequences following damage to human prefrontal cortex , 1994, Cognition.

[37]  H. Sompolinsky,et al.  Theory of orientation tuning in visual cortex. , 1995, Proceedings of the National Academy of Sciences of the United States of America.

[38]  O. Hikosaka Models of information processing in the basal Ganglia edited by James C. Houk, Joel L. Davis and David G. Beiser, The MIT Press, 1995. $60.00 (400 pp) ISBN 0 262 08234 9 , 1995, Trends in Neurosciences.

[39]  S. Nelson,et al.  An emergent model of orientation selectivity in cat visual cortical simple cells , 1995, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[40]  H Eichenbaum,et al.  Information coding in the rodent prefrontal cortex. I. Single-neuron activity in orbitofrontal cortex compared with that in pyriform cortex. , 1995, Journal of neurophysiology.

[41]  A. Barto,et al.  Adaptive Critics and the Basal Ganglia , 1994 .

[42]  G. Buzsáki,et al.  Gamma (40-100 Hz) oscillation in the hippocampus of the behaving rat , 1995, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[43]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[44]  J E Lisman,et al.  Storage of 7 +/- 2 short-term memories in oscillatory subcycles , 1995, Science.

[45]  Richard S. Sutton,et al.  Generalization in ReinforcementLearning : Successful Examples UsingSparse Coarse , 1996 .

[46]  G. Schoenbaum,et al.  Information coding in the rodent prefrontal cortex. II. Ensemble activity in orbitofrontal cortex. , 1995, Journal of neurophysiology.

[47]  David Mumford,et al.  Neuronal Architectures for Pattern-theoretic Problems , 1995 .

[48]  P. Dayan,et al.  A framework for mesencephalic dopamine systems based on predictive Hebbian learning , 1996, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[49]  Geoffrey E. Hinton,et al.  Varieties of Helmholtz Machine , 1996, Neural Networks.

[50]  O Jensen,et al.  Novel lists of 7 +/- 2 known items can be reliably stored in an oscillatory short-term memory network: interaction with long-term memory. , 1996, Learning & memory.

[51]  J. Lisman,et al.  Physiologically realistic formation of autoassociative memory in networks with theta/gamma oscillations: role of fast NMDA channels. , 1996, Learning & memory.

[52]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[53]  A. Alonso,et al.  Muscarinic modulation of the oscillatory and repetitive firing properties of entorhinal cortex layer II neurons. , 1997, Journal of neurophysiology.

[54]  A. Damasio,et al.  Deciding Advantageously Before Knowing the Advantageous Strategy , 1997, Science.

[55]  V. Mountcastle The columnar organization of the neocortex. , 1997, Brain : a journal of neurology.

[56]  D. Johnston,et al.  Regulation of Synaptic Efficacy by Coincidence of Postsynaptic APs and EPSPs , 1997 .

[57]  A. Alonso,et al.  Morphological characteristics of layer II projection neurons in the rat medial entorhinal cortex , 1997, Hippocampus.

[58]  Andrew G. Barto,et al.  Reinforcement learning , 1998 .

[59]  G. Schoenbaum,et al.  Orbitofrontal cortex and basolateral amygdala encode expected outcomes during learning , 1998, Nature Neuroscience.

[60]  B. Balleine,et al.  Goal-directed instrumental action: contingency and incentive learning and their cortical substrates , 1998, Neuropharmacology.

[61]  B. McNaughton,et al.  Firing characteristics of deep layer neurons in prefrontal cortex in rats performing spatial working memory tasks. , 1998, Cerebral cortex.

[62]  G. Bi,et al.  Synaptic Modifications in Cultured Hippocampal Neurons: Dependence on Spike Timing, Synaptic Strength, and Postsynaptic Cell Type , 1998, The Journal of Neuroscience.

[63]  W. Schultz,et al.  Relative reward preference in primate orbitofrontal cortex , 1999, Nature.

[64]  S. Fox,et al.  Action potentials and relations to the theta rhythm of medial septal neurons in vivo , 1999, Experimental Brain Research.

[65]  A. Dickinson,et al.  Neuronal coding of prediction errors. , 2000, Annual review of neuroscience.

[66]  Boris S. Gutkin,et al.  Layer 3 patchy recurrent excitatory connections may determine the spatial organization of sustained activity in the primate prefrontal cortex , 2000, Neurocomputing.

[67]  R. O’Reilly,et al.  Computational Explorations in Cognitive Neuroscience: Understanding the Mind by Simulating the Brain , 2000 .

[68]  M Petrides,et al.  Orbitofrontal cortex: A key prefrontal region for encoding information. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[69]  J. Fuster,et al.  Prefrontal neurons in networks of executive memory , 2000, Brain Research Bulletin.

[70]  Barbara E. Jones,et al.  Discharge Properties of Juxtacellularly Labeled and Immunohistochemically Identified Cholinergic Basal Forebrain Neurons Recorded in Association with the Electroencephalogram in Anesthetized Rats , 2000, The Journal of Neuroscience.

[71]  H. Barbas Connections underlying the synthesis of cognition, memory, and emotion in primate prefrontal cortices , 2000, Brain Research Bulletin.

[72]  J. Hollerman,et al.  Reward processing in primate orbitofrontal cortex and basal ganglia. , 2000, Cerebral cortex.

[73]  W. Schultz,et al.  Reward-related neuronal activity during go-nogo task performance in primate orbitofrontal cortex. , 2000, Journal of neurophysiology.

[74]  W. Schultz,et al.  Modifications of reward expectation-related neuronal activity during learning in primate orbitofrontal cortex. , 2000, Journal of neurophysiology.

[75]  G. Schoenbaum,et al.  Changes in Functional Connectivity in Orbitofrontal Cortex and Basolateral Amygdala during Learning and Reversal Training , 2000, The Journal of Neuroscience.

[76]  E. Miller,et al.  An integrative theory of prefrontal cortex function. , 2001, Annual review of neuroscience.

[77]  S. Grossberg,et al.  A neural model of how horizontal and interlaminar connections of visual cortex develop into adult circuits that carry out perceptual grouping and learning. , 2010, Cerebral cortex.

[78]  Rajesh P. N. Rao,et al.  Spike-Timing-Dependent Hebbian Plasticity as Temporal Difference Learning , 2001, Neural Computation.

[79]  K. C. Anderson,et al.  Single neurons in prefrontal cortex encode abstract rules , 2001, Nature.

[80]  M. Hasselmo,et al.  Simulations of the Role of the Muscarinic-Activated Calcium-Sensitive Nonspecific Cation CurrentINCM in Entorhinal Neuronal Activity during Delayed Matching Tasks , 2002, The Journal of Neuroscience.

[81]  W. Gerstner Integrate-and-Fire Neurons and Networks , 2002 .

[82]  P. Goldman-Rakic,et al.  Correlated discharges among putative pyramidal neurons and interneurons in the primate prefrontal cortex. , 2002, Journal of neurophysiology.

[83]  Michael E. Hasselmo,et al.  A Proposed Function for Hippocampal Theta Rhythm: Separate Phases of Encoding and Retrieval Enhance Reversal of Prior Learning , 2002, Neural Computation.

[84]  Wulfram Gerstner,et al.  SPIKING NEURON MODELS Single Neurons , Populations , Plasticity , 2002 .

[85]  Michael E Hasselmo,et al.  From biophysics to behavior , 2003, Neuroinformatics.

[86]  Michael E. Hasselmo,et al.  Modeling goal-directed spatial navigation in the rat based on physiological data from the hippocampal formation , 2003, Neural Networks.

[87]  E. Miller,et al.  Neuronal activity in primate dorsolateral and orbital prefrontal cortex during performance of a reward preference task , 2003, The European journal of neuroscience.

[88]  Cyriel M. A. Pennartz,et al.  Learning-related changes in response patterns of prefrontal neurons during instrumental conditioning , 2003, Behavioural Brain Research.

[89]  J. Grafman,et al.  Human prefrontal cortex: processing and representational perspectives , 2003, Nature Reviews Neuroscience.

[90]  B. Everitt,et al.  Lesions of the Orbitofrontal but not Medial Prefrontal Cortex Disrupt Conditioned Reinforcement in Primates , 2003, The Journal of Neuroscience.

[91]  J. Parkinson,et al.  Dissociable Contributions of the Human Amygdala and Orbitofrontal Cortex to Incentive Motivation and Goal Selection , 2003, The Journal of Neuroscience.

[92]  Elizabeth M. Brannon,et al.  Serial Expertise of Rhesus Macaques , 2003, Psychological science.

[93]  Geoffrey Schoenbaum,et al.  A systems approach to orbitofrontal cortex function: recordings in rat orbitofrontal cortex reveal interactions with different learning systems , 2003, Behavioural Brain Research.

[94]  Alicia Izquierdo,et al.  Combined unilateral lesions of the amygdala and orbital prefrontal cortex impair affective processing in rhesus monkeys. , 2004, Journal of neurophysiology.

[95]  Jörg Lücke,et al.  Rapid Processing and Unsupervised Learning in a Model of the Cortical Macrocolumn , 2004, Neural Computation.

[96]  D. Mumford On the computational architecture of the neocortex , 2004, Biological Cybernetics.

[97]  D. Mumford,et al.  On the computational architecture of the neocortex , 2004, Biological Cybernetics.

[98]  E. Rolls The functions of the orbitofrontal cortex , 1999, Brain and Cognition.

[99]  S. Thorpe,et al.  The orbitofrontal cortex: Neuronal activity in the behaving monkey , 2004, Experimental Brain Research.

[100]  David Mumford,et al.  On the computational architecture of the neocortex , 2004, Biological Cybernetics.

[101]  J. Elman Distributed representations, simple recurrent networks, and grammatical structure , 1991, Machine Learning.

[102]  Michael E. Hasselmo,et al.  A Model of Prefrontal Cortical Mechanisms for Goal-directed Behavior , 2005, Journal of Cognitive Neuroscience.

[103]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[104]  Richard S. Sutton,et al.  Learning to predict by the methods of temporal differences , 1988, Machine Learning.