Strategic Cognitive Sequencing: A Computational Cognitive Neuroscience Approach

We address strategic cognitive sequencing, the “outer loop” of human cognition: how the brain decides what cognitive process to apply at a given moment to solve complex, multistep cognitive tasks. We argue that this topic has been neglected relative to its importance for systematic reasons but that recent work on how individual brain systems accomplish their computations has set the stage for productively addressing how brain regions coordinate over time to accomplish our most impressive thinking. We present four preliminary neural network models. The first addresses how the prefrontal cortex (PFC) and basal ganglia (BG) cooperate to perform trial-and-error learning of short sequences; the next, how several areas of PFC learn to make predictions of likely reward, and how this contributes to the BG making decisions at the level of strategies. The third models address how PFC, BG, parietal cortex, and hippocampus can work together to memorize sequences of cognitive actions from instruction (or “self-instruction”). The last shows how a constraint satisfaction process can find useful plans. The PFC maintains current and goal states and associates from both of these to find a “bridging” state, an abstract plan. We discuss how these processes could work together to produce strategic cognitive sequencing and discuss future directions in this area.

[1]  Sher ry Folsom-Meek,et al.  Human Performance , 1953, Nature.

[2]  W. Edwards Behavioral decision theory. , 1961, Annual review of psychology.

[3]  J. Fuster,et al.  Reactivity of limbic neurons of the monkey to appetitive and aversive signals. , 1971, Electroencephalography and clinical neurophysiology.

[4]  Earl D. Sacerdoti,et al.  Planning in a Hierarchy of Abstraction Spaces , 1974, IJCAI.

[5]  Allen Newell,et al.  Human Problem Solving. , 1973 .

[6]  A. Newell You can't play 20 questions with nature and win : projective comments on the papers of this symposium , 1973 .

[7]  W. Chase,et al.  Visual information processing. , 1974 .

[8]  L. Beach,et al.  A Contingency Model for the Selection of Decision Strategies , 1978 .

[9]  T. Shallice Specific impairments of planning. , 1982, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[10]  Richard S. Sutton,et al.  Temporal credit assignment in reinforcement learning , 1984 .

[11]  G. E. Alexander,et al.  Parallel organization of functionally segregated circuits linking basal ganglia and cortex. , 1986, Annual review of neuroscience.

[12]  K. Nakamura,et al.  Hypothalamic neuron involvement in integration of reward, aversion, and cue signals. , 1986, Journal of neurophysiology.

[13]  P. Langley,et al.  Production system models of learning and development , 1987 .

[14]  L. Smith,et al.  A model of perceptual classification in children and adults. , 1989, Psychological review.

[15]  Richard S. Sutton,et al.  Time-Derivative Models of Pavlovian Reinforcement , 1990 .

[16]  John R. Anderson,et al.  Rules of the Mind , 1993 .

[17]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[18]  Garrison W. Cottrell,et al.  A Connectionist Model Of Instruction Following , 1995 .

[19]  Avrim Blum,et al.  Fast Planning Through Planning Graph Analysis , 1995, IJCAI.

[20]  Eugene Fink,et al.  Formalizing the PRODIGY planning algorithm , 1996 .

[21]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[22]  A. Owen Tuning in to the temporal dynamics of brain activation using functional magnetic resonance imaging (fMRI) , 1997, Trends in Cognitive Sciences.

[23]  Sebastian Thrun,et al.  Learning to Learn: Introduction and Overview , 1998, Learning to Learn.

[24]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[25]  G. Mangun,et al.  Successful Verbal Encoding into Episodic Memory Engages the Posterior Hippocampus: A Parametrically Analyzed Functional Magnetic Resonance Imaging Study , 1998, The Journal of Neuroscience.

[26]  A. Dagher,et al.  Mapping the network for planning: a correlational PET activation study with the Tower of London task. , 1999, Brain : a journal of neurology.

[27]  R. O’Reilly,et al.  Computational Explorations in Cognitive Neuroscience: Understanding the Mind by Simulating the Brain , 2000 .

[28]  E. Miller,et al.  THE PREFRONTAL CORTEX AND COGNITIVE CONTROL , 2000 .

[29]  E. Miller,et al.  The prefontral cortex and cognitive control , 2000, Nature Reviews Neuroscience.

[30]  Michael J. Frank,et al.  Interactions between frontal cortex and basal ganglia in working memory: A computational model , 2001, Cognitive, affective & behavioral neuroscience.

[31]  Subbarao Kambhampati,et al.  Planning as constraint satisfaction: Solving the planning graph by compiling it into CSP , 2001, Artif. Intell..

[32]  A. Dagher,et al.  The role of the striatum and hippocampus in planning: a PET activation study in Parkinson's disease. , 2001, Brain : a journal of neurology.

[33]  Maxwell J. Roberts,et al.  Understanding strategy selection , 2001, Int. J. Hum. Comput. Stud..

[34]  Christian Lebiere,et al.  To appear in the Proceedings of the Tenth Conference on Computer Generated Forces and Behavioral Representation 1 Multi-Tasking and Cognitive Workload in an ACT-R Model of a Simplified Air Traffic Control Task , 2001 .

[35]  Fahiem Bacchus,et al.  AIPS 2000 Planning Competition: The Fifth International Conference on Artificial Intelligence Planning and Scheduling Systems , 2001 .

[36]  P. Dayan,et al.  Reward, Motivation, and Reinforcement Learning , 2002, Neuron.

[37]  Eytan Ruppin,et al.  Actor-critic models of the basal ganglia: new anatomical and computational perspectives , 2002, Neural Networks.

[38]  Karl J. Friston,et al.  Temporal Difference Models and Reward-Related Learning in the Human Brain , 2003, Neuron.

[39]  Frederik Barkhof,et al.  Frontostriatal system in planning complexity: a parametric functional magnetic resonance version of tower of london task , 2003, NeuroImage.

[40]  C. Ruff,et al.  The Tower of London: the impact of instructions, cueing, and learning on planning abilities. , 2003, Brain research. Cognitive brain research.

[41]  Jonathan Baxter,et al.  A Bayesian/Information Theoretic Model of Learning to Learn via Multiple Task Sampling , 1997, Machine Learning.

[42]  R. Morris,et al.  The Cognitive Psychology of Planning , 2004 .

[43]  R. Hampson,et al.  Reward, memory and substance abuse: functional neuronal circuits in the nucleus accumbens , 2004, Neuroscience & Biobehavioral Reviews.

[44]  Michael J. Frank,et al.  By Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism , 2004, Science.

[45]  John R Anderson,et al.  An integrated theory of the mind. , 2004, Psychological review.

[46]  James L. McClelland,et al.  Semantic Cognition: A Parallel Distributed Processing Approach , 2004 .

[47]  Michael J. Frank,et al.  Dynamic Dopamine Modulation in the Basal Ganglia: A Neurocomputational Account of Cognitive Deficits in Medicated and Nonmedicated Parkinsonism , 2005, Journal of Cognitive Neuroscience.

[48]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[49]  Thomas E. Hazy,et al.  Banishing the homunculus: Making working memory work , 2006, Neuroscience.

[50]  P. Redgrave,et al.  The short-latency dopamine signal: a role in discovering novel actions? , 2006, Nature Reviews Neuroscience.

[51]  H. Yin,et al.  The role of the basal ganglia in habit formation , 2006, Nature Reviews Neuroscience.

[52]  Michael J. Frank,et al.  Making Working Memory Work: A Computational Model of Learning in the Prefrontal Cortex and Basal Ganglia , 2006, Neural Computation.

[53]  S. Mahadevan,et al.  Proto-transfer Learning in Markov Decision Processes Using Spectral Methods , 2006 .

[54]  M. Frank,et al.  Anatomy of a decision: striato-orbitofrontal interactions in reinforcement learning, decision making, and reversal. , 2006, Psychological review.

[55]  Andrew G. Barto,et al.  Building Portable Options: Skill Transfer in Reinforcement Learning , 2007, IJCAI.

[56]  J. Wallis Orbitofrontal cortex and its contribution to decision-making. , 2007, Annual review of neuroscience.

[57]  Thomas E. Hazy,et al.  PVLV: the primary value and learned value Pavlovian learning algorithm. , 2007, Behavioral neuroscience.

[58]  J. Tanji,et al.  Categorization of behavioural sequences in the prefrontal cortex , 2007, Nature.

[59]  John R. Anderson How Can the Human Mind Occur in the Physical Universe , 2007 .

[60]  Thomas E. Hazy,et al.  Towards an executive without a homunculus: computational models of the prefrontal cortex/basal ganglia system , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[61]  Peter Dayan,et al.  Bilinearity, Rules, and Prefrontal Cortex , 2007, Frontiers Comput. Neurosci..

[62]  K. Sakai Task set and prefrontal cortex. , 2008, Annual review of neuroscience.

[63]  J. Tanji,et al.  Role of the lateral prefrontal cortex in executive behavioral control. , 2008, Physiological reviews.

[64]  John R. Anderson,et al.  SAL: an explicitly pluralistic cognitive architecture , 2008, J. Exp. Theor. Artif. Intell..

[65]  John R. Anderson,et al.  Solving the credit assignment problem: explicit and implicit learning of action sequences with probabilistic outcomes , 2008, Psychological research.

[66]  D. Hassabis,et al.  Tracking the Emergence of Conceptual Knowledge during Human Decision Making , 2009, Neuron.

[67]  M. Frank,et al.  Instructional control of reinforcement learning: A behavioral and neurocomputational investigation , 2009, Brain Research.

[68]  Timothy Edward John Behrens,et al.  Effort-Based Cost–Benefit Valuation and the Human Brain , 2009, The Journal of Neuroscience.

[69]  Richard Gonzalez,et al.  Computational Models for the Combination of Advice and Individual Learning , 2009, Cogn. Sci..

[70]  M. Delgado,et al.  How instructed knowledge modulates the neural systems of reward learning , 2010, Proceedings of the National Academy of Sciences.

[71]  Wolfgang M. Pauli,et al.  Computational models of cognitive control , 2010, Current Opinion in Neurobiology.

[72]  Kurt VanLehn,et al.  Meta-Cognitive Strategy Instruction in Intelligent Tutoring Systems: How, When, and Why , 2010, J. Educ. Technol. Soc..

[73]  Joshua L. Jones,et al.  Phasic Nucleus Accumbens Dopamine Release Encodes Effort- and Delay-Related Costs , 2010, Biological Psychiatry.

[74]  Maria Fox,et al.  Constraint Based Planning with Composable Substate Graphs , 2010, ECAI.

[75]  Thomas E. Hazy,et al.  Neural mechanisms of acquired phasic dopamine responses in learning , 2010, Neuroscience & Biobehavioral Reviews.

[76]  Ming-Chou Liu,et al.  Investigating Knowledge Integration in Web-based Thematic Learning Using Concept Mapping Assessment , 2010, J. Educ. Technol. Soc..

[77]  S. Shettleworth Clever animals and killjoy explanations in comparative psychology , 2010, Trends in Cognitive Sciences.

[78]  M. Corballis,et al.  Behavioural evidence for mental time travel in nonhuman animals , 2010, Behavioural Brain Research.

[79]  Seth A. Herd,et al.  From an Executive Network to Executive Control: A Computational Model of the n-back Task , 2011, Journal of Cognitive Neuroscience.

[80]  John R. Anderson,et al.  Modulation of the feedback-related negativity by instruction and experience , 2011, Proceedings of the National Academy of Sciences.

[81]  M. Walton,et al.  Re‐evaluating the role of the orbitofrontal cortex in reward and reinforcement , 2012, The European journal of neuroscience.

[82]  Seth A. Herd,et al.  The Leabra Cognitive Architecture: How to Play 20 Principles with Nature and Win! , 2012 .

[83]  D. Durstewitz,et al.  Contextual encoding by ensembles of medial prefrontal cortex neurons , 2012, Proceedings of the National Academy of Sciences.

[84]  J. Duncan,et al.  Task rules, working memory, and fluid intelligence , 2012, Psychonomic bulletin & review.

[85]  David J. Jilk,et al.  Recurrent Processing during Object Recognition , 2011, Front. Psychol..