Making Working Memory Work: A Computational Model of Learning in the Prefrontal Cortex and Basal Ganglia

The prefrontal cortex has long been thought to subserve both working memory (the holding of information online for processing) and executive functions (deciding how to manipulate working memory and perform processing). Although many computational models of working memory have been developed, the mechanistic basis of executive function remains elusive, often amounting to a homunculus. This article presents an attempt to deconstruct this homunculus through powerful learning mechanisms that allow a computational model of the prefrontal cortex to control both itself and other brain areas in a strategic, task-appropriate manner. These learning mechanisms are based on subcortical structures in the midbrain, basal ganglia, and amygdala, which together form an actor-critic architecture. The critic system learns which prefrontal representations are task relevant and trains the actor, which in turn provides a dynamic gating mechanism for controlling working memory updating. Computationally, the learning mechanism is designed to simultaneously solve the temporal and structural credit assignment problems. The model's performance compares favorably with standard backpropagation-based temporal learning mechanisms on the challenging 1-2-AX working memory task and other benchmark working memory tasks.

[1]  H. Niki,et al.  Prefrontal cortical unit activity and delayed alternation performance in monkeys. , 1971, Journal of neurophysiology.

[2]  G. E. Alexander,et al.  Neuron Activity Related to Short-Term Memory , 1971, Science.

[3]  R. Rescorla,et al.  A theory of Pavlovian conditioning : Variations in the effectiveness of reinforcement and nonreinforcement , 1972 .

[4]  C. W. Ragsdale,et al.  Histochemically distinct compartments in the striatum of human, monkeys, and cat demonstrated by acetylthiocholinesterase staining. , 1978, Proceedings of the National Academy of Sciences of the United States of America.

[5]  E. Oja Simplified neuron model as a principal component analyzer , 1982, Journal of mathematical biology.

[6]  G. E. Alexander,et al.  Parallel organization of functionally segregated circuits linking basal ganglia and cortex. , 1986, Annual review of neuroscience.

[7]  Bernard Widrow,et al.  Adaptive switching circuits , 1988 .

[8]  Y. Miyashita,et al.  Neuronal correlate of pictorial short-term memory in the primate temporal cortexYasushi Miyashita , 1988, Nature.

[9]  Geoffrey E. Hinton Deterministic Boltzmann Learning Performs Steepest Descent in Weight-Space , 1989, Neural Computation.

[10]  P. Goldman-Rakic,et al.  Mnemonic coding of visual space in the monkey's dorsolateral prefrontal cortex. , 1989, Journal of neurophysiology.

[11]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[12]  Michael I. Jordan Attractor dynamics and parallelism in a connectionist sequential machine , 1990 .

[13]  Jürgen Schmidhuber,et al.  Learning Unambiguous Reduced Sequence Descriptions , 1991, NIPS.

[14]  Javier R. Movellan,et al.  Contrastive Hebbian Learning in the Continuous Hopfield Model , 1991 .

[15]  W. Schultz,et al.  Neuronal activity in monkey ventral striatum related to the expectation of reward , 1992, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[16]  D. Surmeier,et al.  D1 and D2 dopamine receptor modulation of sodium and potassium currents in rat neostriatal neurons. , 1993, Progress in brain research.

[17]  J. Wickens A Theory of the Striatum , 1993 .

[18]  J. B. Levitt,et al.  Topography of pyramidal neuron intrinsic connections in macaque monkey prefrontal cortex (areas 9 and 46) , 1993, The Journal of comparative neurology.

[19]  W. Schultz,et al.  Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task , 1993, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[20]  Joel L. Davis,et al.  A Model of How the Basal Ganglia Generate and Use Neural Signals That Predict Reinforcement , 1994 .

[21]  C. Wilson,et al.  Spontaneous firing patterns and axonal projections of single corticostriatal neurons in the rat medial agranular cortex. , 1994, Journal of neurophysiology.

[22]  Joel L. Davis,et al.  Sensorimotor Selection and the Basal Ganglia: A Neural Network Model , 1994 .

[23]  C. Pennartz,et al.  The nucleus accumbens as a complex of functionally distinct neuronal ensembles: An integration of behavioural, electrophysiological and anatomical data , 1994, Progress in Neurobiology.

[24]  Charles J. Wilson,et al.  Surround inhibition among projection neurons is weak or nonexistent in the rat neostriatum. , 1994, Journal of neurophysiology.

[25]  Ronald J. Williams,et al.  Gradient-based learning algorithms for recurrent networks and their computational complexity , 1995 .

[26]  O. Hikosaka Models of information processing in the basal Ganglia edited by James C. Houk, Joel L. Davis and David G. Beiser, The MIT Press, 1995. $60.00 (400 pp) ISBN 0 262 08234 9 , 1995, Trends in Neurosciences.

[27]  James L. McClelland,et al.  Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. , 1995, Psychological review.

[28]  Peter Ford Dominey,et al.  A Model of Corticostriatal Plasticity for Learning Oculomotor Associations and Sequences , 1995, Journal of Cognitive Neuroscience.

[29]  A. Graybiel,et al.  Adaptive neural networks in the basal ganglia. , 1995 .

[30]  J. Wickens,et al.  Effects of local connectivity on striatal function: Simulation and analysis of a model , 1995, Synapse.

[31]  S P Wise,et al.  Distributed modular architectures linking basal ganglia, cerebellum, and cerebral cortex: their role in planning and controlling action. , 1995, Cerebral cortex.

[32]  A. Dickinson,et al.  Reward-related signals carried by dopamine neurons. , 1995 .

[33]  T. Sejnowski,et al.  How the Basal Ganglia Make Decisions , 1996 .

[34]  P. Dayan,et al.  A framework for mesencephalic dopamine systems based on predictive Hebbian learning , 1996, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[35]  J. Mink THE BASAL GANGLIA: FOCUSED SELECTION AND INHIBITION OF COMPETING MOTOR PROGRAMS , 1996, Progress in Neurobiology.

[36]  J. B. Levitt,et al.  Patterns of intrinsic and associational circuitry in monkey prefrontal cortex , 1996, The Journal of comparative neurology.

[37]  R. O’Reilly,et al.  A computational approach to prefrontal cortex, cognitive control and schizophrenia: recent developments and current challenges. , 1996, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[38]  J. Cohen,et al.  Schizophrenic deficits in the processing of context. A test of a theoretical model. , 1996, Archives of general psychiatry.

[39]  Randall C. O'Reilly,et al.  Biologically Plausible Error-Driven Learning Using Local Activation Differences: The Generalized Recirculation Algorithm , 1996, Neural Computation.

[40]  R. Desimone,et al.  Neural Mechanisms of Visual Working Memory in Prefrontal Cortex of the Macaque , 1996, The Journal of Neuroscience.

[41]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[42]  J. Bargas,et al.  D1 Receptor Activation Enhances Evoked Discharge in Neostriatal Medium Spiny Neurons by Modulating an L-Type Ca2+ Conductance , 1997, The Journal of Neuroscience.

[43]  S. Smith‐Roe,et al.  Response-reinforcement learning is dependent on N-methyl-D-aspartate receptor activation in the nucleus accumbens core. , 1997, Proceedings of the National Academy of Sciences of the United States of America.

[44]  L. Abbott,et al.  Synaptic Depression and Cortical Gain Control , 1997, Science.

[45]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[46]  Charles J. Wilson,et al.  Spontaneous subthreshold membrane potential fluctuations and action potential variability of rat corticostriatal and striatal neurons in vivo. , 1997, Journal of neurophysiology.

[47]  Jonathan D. Cohen,et al.  Dissociating working memory from task difficulty in human prefrontal cortex , 1997, Neuropsychologia.

[48]  J. Bargas,et al.  D 1 Receptor Activation Enhances Evoked Discharge in Neostriatal Medium Spiny Neurons by Modulating an L-Type Ca 2 1 Conductance , 1997 .

[49]  C. Lebiere,et al.  The Atomic Components of Thought , 1998 .

[50]  A. Baddeley,et al.  The phonological loop as a language learning device. , 1998, Psychological review.

[51]  J. Fellous,et al.  A role for NMDA-receptor channels in working memory , 1998, Nature Neuroscience.

[52]  T. Sejnowski,et al.  A Computational Model of How the Basal Ganglia Produce Sequences , 1998, Journal of Cognitive Neuroscience.

[53]  R. O’Reilly Six principles for biologically based computational models of cortical cognition , 1998, Trends in Cognitive Sciences.

[54]  P. Stratta,et al.  Schizophrenic deficits in the processing of context. , 1998, Archives of general psychiatry.

[55]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[56]  J. Houk,et al.  Model of cortical-basal ganglionic processing: encoding the serial order of sensory events. , 1998, Journal of neurophysiology.

[57]  D. Plenz,et al.  Up and Down States in Striatal Medium Spiny Neurons Simultaneously Recorded with Spontaneous Activity in Fast-Spiking Interneurons Studied in Cortex–Striatum–Substantia Nigra Organotypic Cultures , 1998, The Journal of Neuroscience.

[58]  J. Kropotov,et al.  Selection of actions in the basal ganglia-thalamocortical circuits: review and model. , 1999, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[59]  A. Charara,et al.  Pre- and postsynaptic localization of GABAB receptors in the basal ganglia in monkeys , 1999, Neuroscience.

[60]  Boris S. Gutkin,et al.  Effects of dopaminergic modulation of persistent sodium currents on the excitability of prefrontal cortical neurons: A computational study , 1999, Neurocomputing.

[61]  Jonathan D. Cohen,et al.  A Biologically Based Computational Model of Working Memory , 1999 .

[62]  R. F. Mayer The Prefrontal Cortex: Anatomy, Physiology and Neuropsychology of the Frontal Lobe, 3rd Edition. , 1999 .

[63]  A. Miyake,et al.  Models of Working Memory: Mechanisms of Active Maintenance and Executive Control , 1999 .

[64]  X. Wang,et al.  Synaptic Basis of Cortical Persistent Activity: the Importance of NMDA Receptors to Working Memory , 1999, The Journal of Neuroscience.

[65]  J. Cohen,et al.  Context-processing deficits in schizophrenia: converging evidence from three theoretically motivated cognitive tasks. , 1999, Journal of abnormal psychology.

[66]  D. Durstewitz,et al.  A Neurocomputational Theory of the Dopaminergic Modulation of Working Memory Functions , 1999, The Journal of Neuroscience.

[67]  N. Burgess Memory for Serial Order : A Network Model of the Phonological Loop and its Timing , 1999 .

[68]  Jürgen Schmidhuber,et al.  Learning to Forget: Continual Prediction with LSTM , 2000, Neural Computation.

[69]  P. Strick,et al.  Basal ganglia and cerebellar loops: motor and cognitive circuits , 2000, Brain Research Reviews.

[70]  T. Sejnowski,et al.  Dopamine-mediated stabilization of delay-period activity in a network model of prefrontal cortex. , 2000, Journal of neurophysiology.

[71]  A. Amos A Computational Model of Information Processing in the Frontal Cortex and Basal Ganglia , 2000, Journal of Cognitive Neuroscience.

[72]  C. Gerfen Molecular effects of dopamine on striatal-projection pathways , 2000, Trends in Neurosciences.

[73]  R. O’Reilly,et al.  Computational Explorations in Cognitive Neuroscience: Understanding the Mind by Simulating the Brain , 2000 .

[74]  T. Braver Working Memory , Cognitive Control , and the Prefrontal Cortex : Computational and Empirical Studies , 2000 .

[75]  E. Koechlin,et al.  Dissociating the role of the medial and lateral anterior prefrontal cortex in human planning. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[76]  J. Bargas,et al.  D2 Dopamine Receptors in Striatal Medium Spiny Neurons Reduce L-Type Ca2+ Currents and Excitability via a Novel PLCβ1–IP3–Calcineurin-Signaling Cascade , 2000, The Journal of Neuroscience.

[77]  N. Gorelova,et al.  Dopamine D1/D5 receptor activation modulates a persistent sodium current in rat prefrontal cortical neurons in vitro. , 2000, Journal of neurophysiology.

[78]  T. Sejnowski,et al.  Neurocomputational models of working memory , 2000, Nature Neuroscience.

[79]  R. Malenka,et al.  Dopaminergic modulation of neuronal excitability in the striatum and nucleus accumbens. , 2000, Annual review of neuroscience.

[80]  D. Joel,et al.  The connections of the dopaminergic system with the striatum in rats and primates: an analysis with respect to the functional and compartmental organization of the striatum , 2000, Neuroscience.

[81]  Nikolaus R. McFarland,et al.  Striatonigrostriatal Pathways in Primates Form an Ascending Spiral from the Shell to the Dorsolateral Striatum , 2000, The Journal of Neuroscience.

[82]  Michael J. Frank,et al.  Interactions between frontal cortex and basal ganglia in working memory: A computational model , 2001, Cognitive, affective & behavioral neuroscience.

[83]  J. Yesavage,et al.  Context processing in older adults: evidence for a theory relating cognitive control to neurobiology in healthy aging. , 2001, Journal of experimental psychology. General.

[84]  J. Cohen,et al.  Selective deficits in prefrontal cortex function in medication-naive patients with schizophrenia. , 2001, Archives of general psychiatry.

[85]  Randall C. O'Reilly,et al.  A Model of the Phonological Loop: Generalization and Binding , 2001, NIPS.

[86]  K. Doya,et al.  Parallel Cortico-Basal Ganglia Mechanisms for Acquisition and Execution of Visuomotor SequencesA Computational Approach , 2001, Journal of Cognitive Neuroscience.

[87]  M. Arbib,et al.  Modeling functions of striatal dopamine modulation in learning and planning , 2001, Neuroscience.

[88]  Randall C. O'Reilly,et al.  Generalization in Interactive Networks: The Benefits of Inhibitory Competition and Hebbian Learning , 2001, Neural Computation.

[89]  Jonathan D. Cohen,et al.  Context processing in older adults: evidence for a theory relating cognitive control to neurobiology in healthy aging. , 2001 .

[90]  Nicolas P. Rougier,et al.  Learning representations in a gated prefrontal cortex model of dynamic task switching , 2002, Cogn. Sci..

[91]  T. Braver,et al.  The Role of Frontopolar Cortex in Subgoal Processing during Working Memory , 2002, NeuroImage.

[92]  Jonathan D. Cohen,et al.  Prefrontal cortex and dynamic categorization tasks: representational organization and neuromodulatory control. , 2002, Cerebral cortex.

[93]  Eytan Ruppin,et al.  Actor-critic models of the basal ganglia: new anatomical and computational perspectives , 2002, Neural Networks.

[94]  A. Kelley,et al.  Early consolidation of instrumental learning requires protein synthesis in the nucleus accumbens , 2002, Nature Neuroscience.

[95]  Y. Munakata,et al.  Active versus latent representations: a neural network model of perseveration, dissociation, and decalage. , 2002, Developmental psychobiology.

[96]  B. Everitt,et al.  Emotion and motivation: the role of the amygdala, ventral striatum, and prefrontal cortex , 2002, Neuroscience & Biobehavioral Reviews.

[97]  M. J. Emerson,et al.  The role of inner speech in task switching: A dual-task investigation , 2003 .

[98]  José Luis Contreras-Vidal,et al.  A Predictive Reinforcement Model of Dopamine Neurons for Learning Approach Behavior , 1999, Journal of Computational Neuroscience.

[99]  Michael J. Frank,et al.  By Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism , 2004, Science.

[100]  John R Anderson,et al.  An integrated theory of the mind. , 2004, Psychological review.

[101]  G. Miller Learning to Forget , 2004, Science.

[102]  Jonathan D. Cohen,et al.  Prefrontal cortex and flexible cognitive control: rules without symbols. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[103]  Michael J. Frank,et al.  Dynamic Dopamine Modulation in the Basal Ganglia: A Neurocomputational Account of Cognitive Deficits in Medicated and Nonmedicated Parkinsonism , 2005, Journal of Cognitive Neuroscience.

[104]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[105]  Richard S. Sutton,et al.  Learning to predict by the methods of temporal differences , 1988, Machine Learning.

[106]  Michael J. Frank,et al.  A mechanistic account of striatal dopamine function in human cognition: psychopharmacological studies with cabergoline and haloperidol. , 2006, Behavioral neuroscience.

[107]  Michael J. Frank,et al.  Hold your horses: A dynamic computational role for the subthalamic nucleus in decision making , 2006, Neural Networks.

[108]  M. Frank,et al.  Anatomy of a decision: striato-orbitofrontal interactions in reinforcement learning, decision making, and reversal. , 2006, Psychological review.

[109]  J C Houk,et al.  Action selection and refinement in subcortical loops through basal ganglia and cerebellum , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[110]  Vinod Menon,et al.  Temporal dynamics of basal ganglia response and connectivity during verbal working memory , 2007, NeuroImage.

[111]  Thomas E. Hazy,et al.  PVLV: the primary value and learned value Pavlovian learning algorithm. , 2007, Behavioral neuroscience.

[112]  Hoi-Chung Leung,et al.  Load response functions in the human spatial working memory circuit during location memory updating , 2007, NeuroImage.

[113]  R. O’Reilly,et al.  Separate neural substrates for skill learning and performance in the ventral and dorsal striatum , 2007, Nature Neuroscience.

[114]  Michael J. Frank,et al.  Understanding decision-making deficits in neurological conditions: insights from models of natural action selection , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[115]  T. Prescott,et al.  Introduction. Modelling natural action selection , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[116]  Kevin N. Gurney,et al.  The Basal Ganglia and Cortex Implement Optimal Decision Making Between Alternative Actions , 2007, Neural Computation.

[117]  Matthew M Botvinick,et al.  Multilevel structure in behaviour and in the brain: a model of Fuster's hierarchy , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[118]  Thomas E. Hazy,et al.  Towards an executive without a homunculus: computational models of the prefrontal cortex/basal ganglia system , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[119]  Michael J. Frank,et al.  Testing Computational Models of Dopamine and Noradrenaline Dysfunction in Attention Deficit/Hyperactivity Disorder , 2007, Neuropsychopharmacology.

[120]  Jonathan D. Cohen,et al.  On the Control of Control: The Role of Dopamine in Regulating Prefrontal Function and Working Memory , 2007 .

[121]  Benjamin M. Robinson,et al.  Selective Reinforcement Learning Deficits in Schizophrenia Support Predictions from Computational Models of Striatal-Cortical Dysfunction , 2007, Biological Psychiatry.

[122]  Guillén Fernández,et al.  Probing the transformation of discontinuous associations into episodic memory: An event-related fMRI study , 2007, NeuroImage.

[123]  David Badre,et al.  Functional Magnetic Resonance Imaging Evidence for a Hierarchical Organization of the Prefrontal Cortex , 2007, Journal of Cognitive Neuroscience.

[124]  M. Gluck,et al.  Basal ganglia and dopamine contributions to probabilistic category learning , 2008, Neuroscience & Biobehavioral Reviews.

[125]  M. D’Esposito Working memory. , 2008, Handbook of clinical neurology.

[126]  Jonathan D. Cohen,et al.  Prefrontal Cortex and the Flexibility of Cognitive Control : Rules Without Symbols , 2022 .