Cognitive control over learning: creating, clustering, and generalizing task-set structure.

Learning and executive functions such as task-switching share common neural substrates, notably prefrontal cortex and basal ganglia. Understanding how they interact requires studying how cognitive control facilitates learning but also how learning provides the (potentially hidden) structure, such as abstract rules or task-sets, needed for cognitive control. We investigate this question from 3 complementary angles. First, we develop a new context-task-set (C-TS) model, inspired by nonparametric Bayesian methods, specifying how the learner might infer hidden structure (hierarchical rules) and decide to reuse or create new structure in novel situations. Second, we develop a neurobiologically explicit network model to assess mechanisms of such structured learning in hierarchical frontal cortex and basal ganglia circuits. We systematically explore the link between these modeling levels across task demands. We find that the network provides an approximate implementation of high-level C-TS computations, with specific neural mechanisms modulating distinct C-TS parameters. Third, this synergism yields predictions about the nature of human optimal and suboptimal choices and response times during learning and task-switching. In particular, the models suggest that participants spontaneously build task-set structure into a learning problem when not cued to do so, which predicts positive and negative transfer in subsequent generalization tests. We provide experimental evidence for these predictions and show that C-TS provides a good quantitative fit to human sequences of choices. These findings implicate a strong tendency to interactively engage cognitive control and learning, resulting in structured abstract representations that afford generalization opportunities and, thus, potentially long-term rather than short-term optimality.

[1]  J. R. Simon,et al.  Reactions toward the source of stimulation. , 1969, Journal of experimental psychology.

[2]  H. Akaike A new look at the statistical model identification , 1974 .

[3]  K. Stokes Planning for the future of a severely handicapped autistic child , 1977, Journal of autism and childhood schizophrenia.

[4]  D. Aldous Exchangeability and related topics , 1985 .

[5]  G. E. Alexander,et al.  Parallel organization of functionally segregated circuits linking basal ganglia and cortex. , 1986, Annual review of neuroscience.

[6]  C. D. Stern,et al.  Handbook of Chemical Neuroanatomy Methods in Chemical Neuroanatomy. Edited by A. Bjorklund and T. Hokfelt. Elsevier, Amsterdam, 1983. Cloth bound, 548 pp. UK £140. (Volume 1 in the series). , 1986, Neurochemistry International.

[7]  John R. Anderson,et al.  The Adaptive Nature of Human Categorization , 1991 .

[8]  P. Dayan,et al.  A framework for mesencephalic dopamine systems based on predictive Hebbian learning , 1996, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[9]  J. Mink THE BASAL GANGLIA: FOCUSED SELECTION AND INHIBITION OF COMPETING MOTOR PROGRAMS , 1996, Progress in Neurobiology.

[10]  Gregory Ashby,et al.  A neuropsychological theory of multiple systems in category learning. , 1998, Psychological review.

[11]  Colin Camerer,et al.  Experience‐weighted Attraction Learning in Normal Form Games , 1999 .

[12]  R. O’Reilly,et al.  Computational Explorations in Cognitive Neuroscience: Understanding the Mind by Simulating the Brain , 2000 .

[13]  S. Lewandowsky,et al.  Knowledge partitioning: Context-dependent use of expertise , 2000, Memory & cognition.

[14]  Michael J. Frank,et al.  Interactions between frontal cortex and basal ganglia in working memory: A computational model , 2001, Cognitive, affective & behavioral neuroscience.

[15]  James L. McClelland,et al.  The time course of perceptual choice: the leaky, competing accumulator model. , 2001, Psychological review.

[16]  J. Fuster The Prefrontal Cortex—An Update Time Is of the Essence , 2001, Neuron.

[17]  Kenji Doya,et al.  Metalearning and neuromodulation , 2002, Neural Networks.

[18]  David R. Anderson,et al.  Model selection and multimodel inference : a practical information-theoretic approach , 2003 .

[19]  Thomas L. Griffiths,et al.  Hierarchical Topic Models and the Nested Chinese Restaurant Process , 2003, NIPS.

[20]  E. Koechlin,et al.  Supporting On-line Material Materials and Methods , 2022 .

[21]  S. Haber The primate basal ganglia: parallel and integrative networks , 2003, Journal of Chemical Neuroanatomy.

[22]  Peter Dayan,et al.  Inference, Attention, and Decision in a Bayesian Neural Architecture , 2004, NIPS.

[23]  M. Kawato,et al.  Functional Magnetic Resonance Imaging Examination of Two Modular Architectures for Switching Multiple Internal Models , 2004, The Journal of Neuroscience.

[24]  D. Medin,et al.  SUSTAIN: a network model of category learning. , 2004, Psychological review.

[25]  Jonathan D. Cohen,et al.  Prefrontal cortex and flexible cognitive control: rules without symbols. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[26]  James C. Houk,et al.  Agents of the mind , 2005, Biological Cybernetics.

[27]  K. Doya,et al.  Representation of Action-Specific Reward Values in the Striatum , 2005, Science.

[28]  Boris S. Gutkin,et al.  Dopamine modulation in the basal ganglia locks the gate to working memory , 2006, Journal of Computational Neuroscience.

[29]  Michael J. Frank,et al.  Dynamic Dopamine Modulation in the Basal Ganglia: A Neurocomputational Account of Cognitive Deficits in Medicated and Nonmedicated Parkinsonism , 2005, Journal of Cognitive Neuroscience.

[30]  Nachshon Meiran,et al.  Advance task preparation reduces task error rate in the cuing task-switching paradigm , 2005, Memory & cognition.

[31]  R. O’Reilly Biologically Based Computational Models of High-Level Cognition , 2006, Science.

[32]  J. Jonides,et al.  Brain mechanisms of proactive interference in working memory , 2006, Neuroscience.

[33]  Michael J. Frank,et al.  Making Working Memory Work: A Computational Model of Learning in the Prefrontal Cortex and Basal Ganglia , 2006, Neural Computation.

[34]  Thomas L. Griffiths,et al.  A more rational model of categorization , 2006 .

[35]  J. O'Doherty,et al.  The Role of the Ventromedial Prefrontal Cortex in Abstract State-Based Inference during Decision Making in Humans , 2006, The Journal of Neuroscience.

[36]  Kristina M. Visscher,et al.  A Core System for the Implementation of Task Sets , 2006, Neuron.

[37]  Michael J. Frank,et al.  Hold your horses: A dynamic computational role for the subthalamic nucleus in decision making , 2006, Neural Networks.

[38]  Michael I. Jordan,et al.  Hierarchical Dirichlet Processes , 2006 .

[39]  K. Doya,et al.  The computational neurobiology of learning and reward , 2006, Current Opinion in Neurobiology.

[40]  Peter Redgrave,et al.  Basal Ganglia , 2020, Encyclopedia of Autism Spectrum Disorders.

[41]  R. Passingham,et al.  Reading Hidden Intentions in the Human Brain , 2007, Current Biology.

[42]  Michael J. Frank,et al.  Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning , 2007, Proceedings of the National Academy of Sciences.

[43]  Philippe Mailly,et al.  Relationship between the corticostriatal terminals from areas 9 and 46, and those from area 8A, dorsal and rostral premotor cortex and area 24c: an anatomical substrate for cognition to action , 2007, The European journal of neuroscience.

[44]  Jadin C. Jackson,et al.  Reconciling reinforcement learning models with behavioral extinction and renewal: implications for addiction, relapse, and problem gambling. , 2007, Psychological review.

[45]  C. Summerfield,et al.  An information theoretical approach to prefrontal executive function , 2007, Trends in Cognitive Sciences.

[46]  Michael J. Frank,et al.  Understanding decision-making deficits in neurological conditions: insights from models of natural action selection , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[47]  David M. Sobel,et al.  Bayes nets and babies: infants' developing statistical reasoning abilities and their representation of causal knowledge. , 2007, Developmental science.

[48]  Timothy Edward John Behrens,et al.  Triangulating a Cognitive Control Network Using Diffusion-Weighted Magnetic Resonance Imaging (MRI) and Functional MRI , 2007, The Journal of Neuroscience.

[49]  Timothy E. J. Behrens,et al.  Learning the value of information in an uncertain world , 2007, Nature Neuroscience.

[50]  M. D’Esposito,et al.  Impulsive Personality Predicts Dopamine-Dependent Changes in Frontostriatal Activity during Component Processes of Working Memory , 2007, The Journal of Neuroscience.

[51]  R. Bogacz Optimal decision-making theories: linking neurobiology with behaviour , 2007, Trends in Cognitive Sciences.

[52]  K. Sakai Task set and prefrontal cortex. , 2008, Annual review of neuroscience.

[53]  David Badre,et al.  Cognitive control, hierarchy, and the rostro–caudal organization of the frontal lobes , 2008, Trends in Cognitive Sciences.

[54]  Paul R. Schrater,et al.  Structure Learning in Human Sequential Decision-Making , 2008, NIPS.

[55]  Brian Mingus,et al.  The Emergent neural modeling system , 2008, Neural Networks.

[56]  P. Dayan,et al.  Decision theory, reinforcement learning, and the brain , 2008, Cognitive, affective & behavioral neuroscience.

[57]  W. Gaissmaier,et al.  The smart potential behind probability matching , 2008, Cognition.

[58]  O. Monchi,et al.  Dopamine Depletion Impairs Frontostriatal Functional Connectivity during a Set-Shifting Task , 2008, The Journal of Neuroscience.

[59]  Jonathan D. Cohen,et al.  Learning to Use Working Memory in Partially Observable Environments through Dopaminergic Reinforcement , 2008, NIPS.

[60]  M. Botvinick Hierarchical models of behavior and prefrontal function , 2008, Trends in Cognitive Sciences.

[61]  Richard S. J. Frackowiak,et al.  Evidence for Segregated and Integrative Connectivity Patterns in the Human Basal Ganglia , 2008, The Journal of Neuroscience.

[62]  J. Kruschke Bayesian approaches to associative learning: From passive to active learning , 2008, Learning & behavior.

[63]  Michael J. Frank,et al.  A dopaminergic basis for working memory, learning and attentional shifting in Parkinsonism , 2008, Neuropsychologia.

[64]  Jonathan D. Cohen,et al.  Sequential effects: Superstition or rational behavior? , 2008, NIPS.

[65]  O. Hikosaka,et al.  Role for Subthalamic Nucleus Neurons in Switching from Automatic to Controlled Eye Movement , 2008, The Journal of Neuroscience.

[66]  Thomas V. Wiecki,et al.  A neurocomputational account of catalepsy sensitization induced by D2 receptor blockade in rats: context dependency, extinction, and renewal , 2009, Psychopharmacology.

[67]  Jeremy R. Reynolds,et al.  Developing PFC representations using reinforcement learning , 2009, Cognition.

[68]  Karl J. Friston,et al.  Bayesian model selection for group studies , 2009, NeuroImage.

[69]  Finale Doshi-Velez,et al.  The Infinite Partially Observable Markov Decision Process , 2009, NIPS.

[70]  T. Maia Reinforcement learning, conditioning, and the brain: Successes and challenges , 2009, Cognitive, affective & behavioral neuroscience.

[71]  Keiji Tanaka,et al.  Conflict-induced behavioural adjustment: a clue to the executive functions of the prefrontal cortex , 2009, Nature Reviews Neuroscience.

[72]  Ulman Lindenberger,et al.  A Neurocomputational Account , 2009 .

[73]  Schrater Paul Structure learning in human sequential decision-making , 2009 .

[74]  Emmanuel M. Pothos,et al.  Exemplar similarity and rule application , 2010, Cognition.

[75]  D. Blei,et al.  Context, learning, and extinction. , 2010, Psychological review.

[76]  Bradley C. Love,et al.  Direct Associations or Internal Transformations? Exploring the Mechanisms Underlying Sequential Learning Behavior , 2009, Cogn. Sci..

[77]  H. Karnath,et al.  Keeping Memory Clear and Stable—The Contribution of Human Basal Ganglia and Prefrontal Cortex to Working Memory , 2010, The Journal of Neuroscience.

[78]  R. Cools,et al.  The Human Basal Ganglia Modulate Frontal-Posterior Connectivity during Attention Shifting , 2010, The Journal of Neuroscience.

[79]  C. Lebiere,et al.  Conditional routing of information to the cortex: a model of the basal ganglia's role in cognitive coordination. , 2010, Psychological review.

[80]  C S Green,et al.  Alterations in choice behavior by manipulations of world model , 2010, Proceedings of the National Academy of Sciences.

[81]  Adam N Sanborn,et al.  Rational approximations to rational models: alternative algorithms for category learning. , 2010, Psychological review.

[82]  Robert C. Wilson,et al.  An Approximately Bayesian Delta-Rule Model Explains the Dynamics of Belief Updating in a Changing Environment , 2010, The Journal of Neuroscience.

[83]  M. D’Esposito,et al.  Frontal Cortex and the Discovery of Abstract Action Rules , 2010, Neuron.

[84]  K. Richard Ridderinkhof,et al.  The Effect of Parkinson's Disease on the Dynamics of On-line and Proactive Cognitive Control during Action Selection , 2010, Journal of Cognitive Neuroscience.

[85]  John Duncan,et al.  Multi-voxel coding of stimuli, rules, and responses in human frontoparietal cortex , 2011, NeuroImage.

[86]  Samuel J. Gershman,et al.  A Tutorial on Bayesian Nonparametric Models , 2011, 1106.2697.

[87]  A. Nambu Somatotopic Organization of the Primate Basal Ganglia , 2011, Front. Neuroanat..

[88]  Thomas V. Wiecki,et al.  Subthalamic nucleus stimulation reverses mediofrontal influence over decision threshold , 2011, Nature Neuroscience.

[89]  M. Frank,et al.  Probabilistic reinforcement learning in adults with autism spectrum disorders , 2011, Autism research : official journal of the International Society for Autism Research.

[90]  M. Frank,et al.  From reinforcement learning models to psychiatric and neurological disorders , 2011, Nature Neuroscience.

[91]  J. Kruschke Models of Attentional Learning , 2011 .

[92]  Joshua B. Tenenbaum,et al.  A probabilistic model of cross-categorization , 2011, Cognition.

[93]  Roger Ratcliff,et al.  Reinforcement-Based Decision Making in Corticostriatal Circuits: Mutual Constraints by Neurocomputational and Diffusion Models , 2012, Neural Computation.

[94]  M. Frank,et al.  Mechanisms of hierarchical reinforcement learning in cortico-striatal circuits 2: evidence from fMRI. , 2012, Cerebral cortex.

[95]  Robert C. Wilson,et al.  Inferring Relevance in a Changing World , 2012, Front. Hum. Neurosci..

[96]  M. Kahana,et al.  Neuronal Activity in the Human Subthalamic Nucleus Encodes Decision Conflict during Action Selection , 2012, The Journal of Neuroscience.

[97]  M. Frank,et al.  Mechanisms of hierarchical reinforcement learning in corticostriatal circuits 1: computational analysis. , 2012, Cerebral cortex.

[98]  E. Koechlin,et al.  Reasoning, Learning, and Creativity: Frontal Lobe Function and Human Decision-Making , 2012, PLoS biology.

[99]  Nicole M. Long,et al.  Supplemental Figure , 2013 .

[100]  John-Dylan Haynes,et al.  Compositionality of rule representations in human prefrontal cortex. , 2012, Cerebral cortex.

[101]  M. Botvinick Hierarchical reinforcement learning and decision making , 2012, Current Opinion in Neurobiology.

[102]  Anne G E Collins,et al.  How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis , 2012, The European journal of neuroscience.

[103]  G. D. Logan Task Switching , 2022 .