What is a cognitive map? Organising knowledge for flexible behaviour

It is proposed that a cognitive map encoding the relationships between entities in the world supports flexible behaviour, but the majority of the neural evidence for such a system comes from studies of spatial navigation. Recent work describing neuronal parallels between spatial and non-spatial behaviours has rekindled the notion of a systematic organisation of knowledge across multiple domains. We review experimental evidence and theoretical frameworks that point to principles unifying these apparently disparate functions. These principles describe how to learn and use abstract, generalisable knowledge and suggest map-like representations observed in a spatial context may be an instance of general coding mechanisms capable of organising knowledge of all kinds. We highlight how artificial agents endowed with such principles exhibit flexible behaviour and learn map-like representations observed in the brain. Finally, we speculate on how these principles may offer insight into the extreme generalisations, abstractions and inferences that characterise human cognition.

[1]  J. O’Keefe,et al.  Boundary Vector Cells in the Subiculum of the Hippocampal Formation , 2009, The Journal of Neuroscience.

[2]  Marcin Andrychowicz,et al.  Learning to learn by gradient descent by gradient descent , 2016, NIPS.

[3]  Joshua B. Tenenbaum,et al.  Human-level concept learning through probabilistic program induction , 2015, Science.

[4]  Christian F. Doeller,et al.  Evidence for grid cells in a human memory network , 2010, Nature.

[5]  Pablo E. Jercog,et al.  Neural ensemble dynamics underlying a long-term associative memory , 2017, Nature.

[6]  M. Moser,et al.  Representation of Geometric Borders in the Entorhinal Cortex , 2008, Science.

[7]  E. Murray,et al.  The Orbitofrontal Oracle: Cortical Mechanisms for the Prediction and Evaluation of Specific Behavioral Outcomes , 2014, Neuron.

[8]  Timothy Edward John Behrens,et al.  Generalisation of structural knowledge in the Hippocampal-Entorhinal system , 2018, NeurIPS.

[9]  M. Botvinick,et al.  The successor representation in human reinforcement learning , 2016, Nature Human Behaviour.

[10]  H. Eichenbaum,et al.  Robust Conjunctive Item–Place Coding by Hippocampal Neurons Parallels Learning What Happens Where , 2009, The Journal of Neuroscience.

[11]  C. Padoa-Schioppa,et al.  Neurons in the orbitofrontal cortex encode economic value , 2006, Nature.

[12]  Xue-Xin Wei,et al.  Emergence of grid-like representations by training recurrent neural networks to perform spatial localization , 2018, ICLR.

[13]  H. Eichenbaum,et al.  The global record of memory in hippocampal neuronal activity , 1999, Nature.

[14]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[15]  Duncan J. Watts,et al.  Collective dynamics of ‘small-world’ networks , 1998, Nature.

[16]  Anne E Carpenter,et al.  Neuron-type specific signals for reward and punishment in the ventral tegmental area , 2011, Nature.

[17]  Anders Lansner,et al.  Computing the Local Field Potential (LFP) from Integrate-and-Fire Network Models , 2015, PLoS Comput. Biol..

[18]  Andrew M. Wikenheiser,et al.  Over the river, through the woods: cognitive maps in the hippocampus and orbitofrontal cortex , 2016, Nature Reviews Neuroscience.

[19]  Joel Z. Leibo,et al.  Prefrontal cortex as a meta-reinforcement learning system , 2018, bioRxiv.

[20]  Regina Paxton Gazes,et al.  Cognitive mechanisms for transitive inference performance in rhesus monkeys: measuring the influence of associative strength and inferred order. , 2012, Journal of experimental psychology. Animal behavior processes.

[21]  Nathaniel J. Killian,et al.  Grid cells map the visual world , 2017, Nature Neuroscience.

[22]  Matthew T. Kaufman,et al.  A neural network that finds a naturalistic solution for the production of muscle activity , 2015, Nature Neuroscience.

[23]  Timothy E. J. Behrens,et al.  Learning the value of information in an uncertain world , 2007, Nature Neuroscience.

[24]  B. McGonigle,et al.  Are monkeys logical? , 1977, Nature.

[25]  Razvan Pascanu,et al.  Overcoming catastrophic forgetting in neural networks , 2016, Proceedings of the National Academy of Sciences.

[26]  D. Hassabis,et al.  Hippocampal place cells construct reward related sequences through unexplored space , 2015, eLife.

[27]  Sho Yagishita,et al.  A critical time window for dopamine actions on the structural plasticity of dendritic spines , 2014, Science.

[28]  H. Yamahachi,et al.  Hippocampal Remapping after Partial Inactivation of the Medial Entorhinal Cortex , 2015, Neuron.

[29]  R. Buckner,et al.  Self-projection and the brain , 2007, Trends in Cognitive Sciences.

[30]  F. Heider,et al.  An experimental study of apparent behavior , 1944 .

[31]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[32]  H. Eichenbaum,et al.  Evolution of declarative memory , 2006, Hippocampus.

[33]  Peter Dayan,et al.  Interplay of approximate planning strategies , 2015, Proceedings of the National Academy of Sciences.

[34]  Sergey Levine,et al.  Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks , 2017, ICML.

[35]  Nachum Ulanovsky,et al.  Vectorial representation of spatial goals in the hippocampus of bats , 2017, Science.

[36]  I. Fried,et al.  Direct recordings of grid-like neuronal activity in human spatial navigation , 2013, Nature Neuroscience.

[37]  Robert C. Wilson,et al.  Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex , 2011, Nature Neuroscience.

[38]  Vincent P. Ferrera,et al.  Implicit Value Updating Explains Transitive Inference Performance: The Betasort Model , 2015, PLoS Comput. Biol..

[39]  E. Wasserman,et al.  Transitive inference in pigeons: Measuring the associative values of Stimuli B and D , 2012, Behavioural Processes.

[40]  Roddy M. Grieves,et al.  The representation of space in the brain , 2017, Behavioural Processes.

[41]  Jeffrey L. Gauthier,et al.  A Dedicated Population for Reward Coding in the Hippocampus , 2018, Neuron.

[42]  W. Schultz,et al.  Retroactive modulation of spike timing-dependent plasticity by dopamine , 2015, eLife.

[43]  Nathaniel D. Daw,et al.  Grid Cells, Place Cells, and Geodesic Generalization for Spatial Reinforcement Learning , 2011, PLoS Comput. Biol..

[44]  Yoram Burakyy,et al.  Accurate Path Integration in Continuous Attractor Network Models of Grid Cells , 2009 .

[45]  Timothy Edward John Behrens,et al.  Separable Learning Systems in the Macaque Brain and the Role of Orbitofrontal Cortex in Contingent Learning , 2010, Neuron.

[46]  E. Bostock,et al.  Experience‐dependent modifications of hippocampal place cell firing , 1991, Hippocampus.

[47]  R. Passingham.  The hippocampus as a cognitive map J. O'Keefe & L. Nadel, Oxford University Press, Oxford (1978). 570 pp., £25.00 , 1979, Neuroscience.

[48]  Kevin J. Miller,et al.  Value representations in the rodent orbitofrontal cortex drive learning, not choice , 2018, bioRxiv.

[49]  D. Amaral,et al.  Entorhinal Cortex Lesions Disrupt the Relational Organization of Memory in Monkeys , 2004, The Journal of Neuroscience.

[50]  Timothy Edward John Behrens,et al.  The Neural Network Underlying Incentive-Based Learning: Implications for Interpreting Circuit Disruptions in Psychiatric Disorders , 2014, Neuron.

[51]  Charles Kemp,et al.  The discovery of structural form , 2008, Proceedings of the National Academy of Sciences.

[52]  Adam Johnson,et al.  Neural Ensembles in CA3 Transiently Encode Paths Forward of the Animal at a Decision Point , 2007, The Journal of Neuroscience.

[53]  Karl J. Friston,et al.  Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning , 2004, Science.

[54]  Timothy Edward John Behrens,et al.  Two Anatomically and Computationally Distinct Learning Signals Predict Changes to Stimulus-Outcome Associations in Hippocampus , 2016, Neuron.

[55]  Treichler Fr,et al.  Concurrent conditional discrimination tests of transitive inference by macaque monkeys: list linking. , 1996 .

[56]  R U Muller,et al.  Head-direction cells recorded from the postsubiculum in freely moving rats. I. Description and quantitative analysis , 1990, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[57]  Kevin J. Miller,et al.  Dorsal hippocampus contributes to model-based planning , 2017, Nature Neuroscience.

[58]  T. Hafting,et al.  Microstructure of a spatial map in the entorhinal cortex , 2005, Nature.

[59]  L. Nadel,et al.  The Hippocampus as a Cognitive Map , 1978 .

[60]  Timothy Edward John Behrens,et al.  Unmasking Latent Inhibitory Connections in Human Cortex to Reveal Dormant Cortical Memories , 2016, Neuron.

[61]  Tobias Navarro Schröder,et al.  Hexadirectional coding of visual space in human entorhinal cortex , 2018, Nature Neuroscience.

[62]  P. Dayan,et al.  Adaptive integration of habits into depth-limited planning defines a habitual-goal–directed spectrum , 2016, Proceedings of the National Academy of Sciences.

[63]  J. Lisman,et al.  D1/D5 Dopamine Receptor Activation Increases the Magnitude of Early Long-Term Potentiation at CA1 Hippocampal Synapses , 1996, The Journal of Neuroscience.

[64]  J. O’Keefe,et al.  Grid cell firing patterns signal environmental novelty by expansion , 2012, Proceedings of the National Academy of Sciences.

[65]  Fabian Grabenhorst,et al.  A dynamic code for economic object valuation in prefrontal cortex neurons , 2016, Nature Communications.

[66]  E. Murray,et al.  Dissociable Effects of Subtotal Lesions within the Macaque Orbital Prefrontal Cortex on Reward-Guided Behavior , 2011, The Journal of Neuroscience.

[67]  M. Shapiro,et al.  A Map for Social Navigation in the Human Brain , 2015, Neuron.

[68]  Hava T. Siegelmann,et al.  On the Computational Power of Neural Nets , 1995, J. Comput. Syst. Sci..

[69]  Timothy E. J. Behrens,et al.  Review Frontal Cortex and Reward-guided Learning and Decision-making Figure 1. Frontal Brain Regions in the Macaque Involved in Reward-guided Learning and Decision-making Finer Grained Anatomical Divisions with Frontal Cortical Systems for Reward-guided Behavior , 2022 .

[70]  Csaba Szepesvári,et al.  Bandit Based Monte-Carlo Planning , 2006, ECML.

[71]  Andrew M. Wikenheiser,et al.  Suppression of Ventral Hippocampal Output Impairs Integrated Orbitofrontal Encoding of Task Structure , 2017, Neuron.

[72]  Lea Fleischer,et al.  The Senses Considered As Perceptual Systems , 2016 .

[73]  Russell A. Epstein,et al.  Human entorhinal cortex represents visual space using a boundary-anchored grid , 2017, Nature Neuroscience.

[74]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[75]  P. Dayan,et al.  A mathematical model explains saturating axon guidance responses to molecular gradients , 2016, eLife.

[76]  Timothy E.J. Behrens,et al.  Intuitive planning: global navigation through cognitive maps based on grid-like codes , 2018 .

[77]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[78]  Razvan Pascanu,et al.  Vector-based navigation using grid-like representations in artificial agents , 2018, Nature.

[79]  James L. McClelland,et al.  Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. , 1995, Psychological review.

[80]  Sachin S. Deshmukh,et al.  Influence of local objects on hippocampal representations: Landmark vectors and memory , 2013, Hippocampus.

[81]  M. Botvinick,et al.  The hippocampus as a predictive map , 2016 .

[82]  H. Harlow,et al.  The formation of learning sets. , 1949, Psychological review.

[83]  J. Delacour,et al.  Effects of selective lesions of Fimbria-Fornix on learning set in the rat , 1987, Physiology & Behavior.

[84]  Robert C. Wilson,et al.  Orbitofrontal Cortex as a Cognitive Map of Task Space , 2014, Neuron.

[85]  David J. Anderson,et al.  Ventromedial hypothalamic neurons control a defensive emotion state , 2015, eLife.

[86]  P. Dayan,et al.  Cortical substrates for exploratory decisions in humans , 2006, Nature.

[87]  E. Lein,et al.  Functional organization of the hippocampal longitudinal axis , 2014, Nature Reviews Neuroscience.

[88]  F. R. Treichler,et al.  Concurrent conditional discrimination tests of transitive inference by macaque monkeys: list linking. , 1996, Journal of experimental psychology. Animal behavior processes.

[89]  Dmitriy Aronov,et al.  Mapping of a non-spatial dimension by the hippocampal/entorhinal circuit , 2017, Nature.

[90]  B. McNaughton,et al.  Independent Codes for Spatial and Episodic Memory in Hippocampal Neuronal Ensembles , 2005, Science.

[91]  E. Tolman,et al.  Studies in spatial learning: Orientation and the short-cut. , 1946, Journal of experimental psychology.

[92]  H. Eichenbaum,et al.  The hippocampus and memory for orderly stimulus relations. , 1997, Proceedings of the National Academy of Sciences of the United States of America.

[93]  栁下 祥.  A critical time window for dopamine actions on the structural plasticity of dendritic spines , 2016 .

[94]  T. Toyoizumi,et al.  Spatial representations of self and other in the hippocampus , 2018, Science.

[95]  P. Dayan,et al.  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control , 2005, Nature Neuroscience.

[96]  T. Cornsweet,et al.  The staircrase-method in psychophysics. , 1962, The American journal of psychology.

[97]  Camillo Padoa-Schioppa,et al.  Neuronal Remapping and Circuit Persistence in Economic Decisions , 2016, Nature Neuroscience.

[98]  M. Fyhn,et al.  Progressive increase in grid scale from dorsal to ventral medial entorhinal cortex , 2008, Hippocampus.

[99]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[100]  E. Tolman.  Cognitive maps in rats and men. , 1948, Psychological review.

[101]  Surya Ganguli,et al.  Continual Learning Through Synaptic Intelligence , 2017, ICML.

[102]  Edvard I. Moser,et al.  Object-vector coding in the medial entorhinal cortex , 2018, bioRxiv.

[103]  Dylan A. Simon,et al.  Model-based choices involve prospective neural activity , 2015, Nature Neuroscience.

[104]  P. Dayan,et al.  Model-based influences on humans’ choices and striatal prediction errors , 2011, Neuron.

[105]  Matthijs A. A. van der Meer,et al.  Information Processing in Decision-Making Systems , 2012, The Neuroscientist : a review journal bringing neurobiology, neurology and psychiatry.

[106]  Sepp Hochreiter,et al.  Learning to Learn Using Gradient Descent , 2001, ICANN.

[107]  Michael L. Platt,et al.  Neural correlates of decision variables in parietal cortex , 1999, Nature.

[108]  Kenneth A. Norman,et al.  Offline Replay Supports Planning: fMRI Evidence from Reward Revaluation , 2017, bioRxiv.

[109]  Joshua L. Jones,et al.  Orbitofrontal Cortex Supports Behavior and Learning Using Inferred But Not Cached Values , 2012, Science.

[110]  J. O’Keefe,et al.  Neural Representations of Location Composed of Spatially Periodic Bands , 2012, Science.

[111]  G. Winocur,et al.  Higher-Order Conditioning Is Impaired by Hippocampal Lesions , 2014, Current Biology.

[112]  J. Gold,et al.  The neural basis of decision making. , 2007, Annual review of neuroscience.

[113]  Neil Burgess,et al.  Using Grid Cells for Navigation , 2015, Neuron.

[114]  Transitive Inference Formation in Pigeons , 2019 .

[115]  Raymond J Dolan,et al.  A map of abstract relational knowledge in the human hippocampal–entorhinal cortex , 2017, eLife.

[116]  Nicolas W. Schuck,et al.  Human Orbitofrontal Cortex Represents a Cognitive Map of State Space , 2016, Neuron.

[117]  KiJung Yoon,et al.  Grid Cell Responses in 1D Environments Assessed as Slices through a 2D Lattice , 2016, Neuron.

[118]  David Gaffan,et al.  Frontal-temporal disconnection abolishes object discrimination learning set in macaque monkeys. , 2006, Cerebral cortex.

[119]  T. Robbins,et al.  Differential Contributions of the Primate Ventrolateral Prefrontal and Orbitofrontal Cortex to Serial Reversal Learning , 2010, The Journal of Neuroscience.

[120]  N. Ulanovsky,et al.  Social place-cells in the bat hippocampus , 2018, Science.

[121]  Joshua B. Tenenbaum,et al.  Building machines that learn and think like people , 2016, Behavioral and Brain Sciences.

[122]  D. Hassabis,et al.  Patients with hippocampal amnesia cannot imagine new experiences , 2007, Proceedings of the National Academy of Sciences.

[123]  Surya Ganguli,et al.  On the Expressive Power of Deep Neural Networks , 2016, ICML.

[124]  Peter Dayan,et al.  Improving Generalization for Temporal Difference Learning: The Successor Representation , 1993, Neural Computation.

[125]  M. Botvinick,et al.  Neural representations of events arise from temporal community structure , 2013, Nature Neuroscience.

[126]  C. Koch,et al.  Invariant visual representation by single neurons in the human brain , 2005, Nature.

[127]  David S. Lorberbaum,et al.  Genetic evidence that Nkx2.2 acts primarily downstream of Neurog3 in pancreatic endocrine lineage development , 2017, eLife.

[128]  C. H. Honzik,et al.  Degrees of hunger, reward and non-reward, and maze learning in rats, and Introduction and removal of reward, and maze performance in rats , 1930 .

[129]  Razvan Pascanu,et al.  Relational Deep Reinforcement Learning , 2018, ArXiv.

[130]  A. Treves,et al.  Hippocampal remapping and grid realignment in entorhinal cortex , 2007, Nature.

[131]  Ron Meir,et al.  Extracting grid cell characteristics from place cell inputs using non-negative principal component analysis , 2016, eLife.

[132]  J. DiCarlo,et al.  Using goal-driven deep learning models to understand sensory cortex , 2016, Nature Neuroscience.

[133]  Timothy Edward John Behrens,et al.  Reward-Guided Learning with and without Causal Attribution , 2016, Neuron.

[134]  Surya Ganguli,et al.  A Multiplexed, Heterogeneous, and Adaptive Code for Navigation in Medial Entorhinal Cortex , 2017, Neuron.

[135]  Timothy E. J. Behrens,et al.  Organizing conceptual knowledge in humans with a gridlike code , 2016, Science.

[136]  Alexander Mathis,et al.  Connecting multiple spatial scales to decode the population activity of grid cells , 2015, Science Advances.

[137]  Daniel Tranel,et al.  The Human Ventromedial Prefrontal Cortex Is Critical for Transitive Inference , 2012, Journal of Cognitive Neuroscience.

[138]  W. Newsome,et al.  Context-dependent computation by recurrent dynamics in prefrontal cortex , 2013, Nature.