Similarities and differences in spatial and non-spatial cognitive maps

Learning and generalization in spatial domains is often thought to rely on a “cognitive map”, representing relationships between spatial locations. Recent research suggests that this same neural machinery is also recruited for reasoning about more abstract, conceptual forms of knowledge. Yet, to what extent do spatial and conceptual reasoning share common computational principles, and what are the implications for behavior? Using a within-subject design we studied how participants used spatial or conceptual distances to generalize and search for correlated rewards in successive multi-armed bandit tasks. Participant behavior indicated sensitivity to both spatial and conceptual distance, and was best captured using a Bayesian model of generalization that formalized distance-dependent generalization and uncertainty-guided exploration as a Gaussian Process regression with a radial basis function kernel. The same Gaussian Process model best captured human search decisions and judgments in both domains, and could simulate realistic learning curves, where we found equivalent levels of generalization in spatial and conceptual tasks. At the same time, we also find characteristic differences between domains. Relative to the spatial domain, participants showed reduced levels of uncertainty-directed exploration and increased levels of random exploration in the conceptual domain. Participants also displayed a one-directional transfer effect, where experience in the spatial task boosted performance in the conceptual task, but not vice versa. While confidence judgments indicated that participants were sensitive to the uncertainty of their knowledge in both tasks, they did not or could not leverage their estimates of uncertainty to guide exploration in the conceptual task. These results support the notion that value-guided learning and generalization recruit cognitive-map dependent computational mechanisms in spatial and conceptual domains. Yet both behavioral and model-based analyses suggest domain specific differences in how these representations map onto actions.

[1]  Benjamin J. Kraus,et al.  Hippocampal “Time Cells”: Time versus Path Integration , 2013, Neuron.

[2]  Thomas T. Hills,et al.  The central executive as a search process: priming exploration and exploitation across domains. , 2010, Journal of experimental psychology. General.

[3]  E. Wagenmakers,et al.  Harold Jeffreys’s default Bayes factor hypothesis tests: Explanation, extension, and application in psychology , 2016 .

[4]  Jeffrey N. Rouder,et al.  Bayesian t tests for accepting and rejecting the null hypothesis , 2009, Psychonomic bulletin & review.

[5]  Thomas L. Griffiths,et al.  Learning hypothesis spaces and dimensions through concept learning , 2010 .

[6]  Paul-Christian Bürkner,et al.  brms: An R Package for Bayesian Multilevel Models Using Stan , 2017 .

[7]  Thomas T. Hills,et al.  Cognitive search : evolution, algorithms, and the brain , 2012 .

[8]  Nicolas W. Schuck,et al.  Sequential replay of nonspatial task states in the human hippocampus , 2018, Science.

[9]  Gilbert Ritschard,et al.  Coefficient-wise tree-based varying coefficient regression with vcrpart , 2017 .

[10]  Timothy J. Pleskac,et al.  Under pressure: The influence of time limits on human exploration , 2019, CogSci.

[11]  Samuel Gershman,et al.  A Unifying Probabilistic View of Associative Learning , 2015, PLoS Comput. Biol..

[12]  David J. Foster,et al.  Memory and Space: Towards an Understanding of the Cognitive Map , 2015, The Journal of Neuroscience.

[13]  Peter Gärdenfors,et al.  Navigating cognition: Spatial codes for human thinking , 2018, Science.

[14]  M. Moser,et al.  Representation of Geometric Borders in the Entorhinal Cortex , 2008, Science.

[15]  Christopher G. Lucas,et al.  A rational model of function learning , 2015, Psychonomic Bulletin & Review.

[16]  J. Taube Head direction cells and the neurophysiological basis for a sense of direction , 1998, Progress in Neurobiology.

[17]  Samuel Gershman,et al.  Predictive representations can link model-based reinforcement learning to model-free mechanisms , 2017, bioRxiv.

[18]  Øyvind Arne Høydal,et al.  Object-vector coding in the medial entorhinal cortex , 2019, Nature.

[19]  Zeb Kurth-Nelson,et al.  What Is a Cognitive Map? Organizing Knowledge for Flexible Behavior , 2018, Neuron.

[20]  Peter Auer,et al.  Using Confidence Bounds for Exploitation-Exploration Trade-offs , 2003, J. Mach. Learn. Res..

[21]  Martin Rolfs,et al.  Saccadic selection of stabilized items in visuospatial working memory , 2018, Consciousness and Cognition.

[22]  M. Quirk,et al.  Experience-Dependent Asymmetric Shape of Hippocampal Receptive Fields , 2000, Neuron.

[23]  Peter Dayan,et al.  Improving Generalization for Temporal Difference Learning: The Successor Representation , 1993, Neural Computation.

[24]  B. Schölkopf,et al.  Similarity, Kernels, and the Triangle Inequality , 2008 .

[25]  Frances Amelia Yates Art Of Memory , 2013 .

[26]  J. Carroll FUNCTIONAL LEARNING: THE LEARNING OF CONTINUOUS FUNCTIONAL MAPPINGS RELATING STIMULUS AND RESPONSE CONTINUA , 1963 .

[27]  Björn Meder,et al.  Connecting conceptual and spatial search via a model of generalization , 2018, bioRxiv.

[28]  Thomas T. Hills,et al.  Optimal foraging in semantic memory. , 2012, Psychological review.

[29]  Karl J. Friston,et al.  Bayesian model selection for group studies — Revisited , 2014, NeuroImage.

[30]  Dmitriy Aronov,et al.  Mapping of a non-spatial dimension by the hippocampal/entorhinal circuit , 2017, Nature.

[31]  Andreas Krause,et al.  Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting , 2009, IEEE Transactions on Information Theory.

[32]  Maarten Speekenbrink,et al.  Uncertainty and Exploration in a Restless Bandit Problem , 2015, Top. Cogn. Sci..

[33]  Joseph L. Austerweil,et al.  Learning How to Generalize , 2019, Cogn. Sci..

[34]  Bradley C. Love,et al.  A non-spatial account of place and grid cells based on clustering models of concept learning , 2018, Nature Communications.

[35]  Jeffrey N. Rouder,et al.  Default Bayes factors for ANOVA designs , 2012 .

[36]  Karl J. Friston,et al.  Bayesian model selection for group studies , 2009, NeuroImage.

[37]  Ian C. Ballard,et al.  Holistic Reinforcement Learning: The Role of Structure and Attention , 2019, Trends in Cognitive Sciences.

[38]  Joshua B. Tenenbaum,et al.  Assessing the Perceived Predictability of Functions , 2015, CogSci.

[39]  J. O’Keefe A review of the hippocampal place cells , 1979, Progress in Neurobiology.

[40]  Haitao Liu,et al.  When Gaussian Process Meets Big Data: A Review of Scalable GPs , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[41]  Kimberly L. Stachenfeld,et al.  The hippocampus as a predictive map , 2017, Nature Neuroscience.

[42]  Christian F. Doeller,et al.  The Role of Mental Maps in Decision-Making , 2017, Trends in Neurosciences.

[43]  H. Eichenbaum,et al.  Distinct Hippocampal Time Cell Sequences Represent Odor Memories in Immobilized Rats , 2013, The Journal of Neuroscience.

[44]  E. Wagenmakers,et al.  Bayesian Inference for Kendall’s Rank Correlation Coefficient , 2018 .

[45]  J. Hartman,et al.  PAK4 suppresses RELB to prevent senescence-like growth arrest in breast cancer , 2019, Nature Communications.

[46]  B. Landau,et al.  Whence and whither in spatial language and spatial cognition , 1993 .

[47]  David Ardia,et al.  DEoptim: An R Package for Global Optimization by Differential Evolution , 2009 .

[48]  H. Eichenbaum Hippocampus: Remembering the Choices , 2013, Neuron.

[49]  Christian F. Doeller,et al.  Evidence for grid cells in a human memory network , 2010, Nature.

[50]  P. Tobler,et al.  Dopamine regulates stimulus generalization in the human hippocampus , 2016, eLife.

[51]  I. Fried,et al.  Direct recordings of grid-like neuronal activity in human spatial navigation , 2013, Nature Neuroscience.

[52]  D. Barr,et al.  Random effects structure for confirmatory hypothesis testing: Keep it maximal. , 2013, Journal of memory and language.

[53]  Andrew Gelman,et al.  The No-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo , 2011, J. Mach. Learn. Res..

[54]  Tomaso Poggio,et al.  Generalization in vision and motor control , 2004, Nature.

[55]  Samuel J. Gershman,et al.  The algorithmic architecture of exploration in the human brain , 2019, Current Opinion in Neurobiology.

[56]  E. Tolman Cognitive maps in rats and men. , 1948, Psychological review.

[57]  H. Spiers The Hippocampal Cognitive Map: One Space or Many? , 2020, Trends in Cognitive Sciences.

[58]  P. Thorndyke Distance estimation from cognitive maps , 1981, Cognitive Psychology.

[59]  Russell A. Epstein,et al.  The cognitive map in humans: spatial navigation and beyond , 2017, Nature Neuroscience.

[60]  Axel Cleeremans,et al.  Should we control? The interplay between cognitive control and information integration in the resolution of the exploration-exploitation dilemma. , 2019, Journal of experimental psychology. General.

[61]  Thomas T. Hills,et al.  Search in External and Internal Spaces , 2008, Psychological science.

[62]  Matthias Nau,et al.  Deforming the metric of cognitive maps distorts memory , 2019, Nature Human Behaviour.

[63]  Jonathan D. Cohen,et al.  Humans use directed and random exploration to solve the explore-exploit dilemma. , 2014, Journal of experimental psychology. General.

[64]  Matthew A Wilson,et al.  Hippocampal remapping as hidden state inference , 2020, eLife.

[65]  Timothy E. J. Behrens,et al.  Organizing conceptual knowledge in humans with a gridlike code , 2016, Science.

[66]  D. Meyer,et al.  Function learning: induction of continuous stimulus-response relations. , 1991, Journal of experimental psychology. Learning, memory, and cognition.

[67]  A. Zellner,et al.  Posterior odds ratios for selected regression hypotheses , 1980 .

[68]  Jonathan D. Nelson,et al.  Generalization guides human exploration in vast decision spaces , 2017, Nature Human Behaviour.

[69]  S. Gershman Deconstructing the human algorithms for exploration , 2018, Cognition.

[70]  Raymond J Dolan,et al.  A map of abstract relational knowledge in the human hippocampal–entorhinal cortex , 2017, eLife.

[71]  Yael Niv,et al.  A State Representation for Reinforcement Learning and Decision-Making in the Orbitofrontal Cortex , 2018 .

[72]  Nicolas W. Schuck,et al.  Human Orbitofrontal Cortex Represents a Cognitive Map of State Space , 2016, Neuron.

[73]  V. Wyart,et al.  Computational noise in reward-guided learning drives behavioral variability in volatile environments , 2019, Nature Neuroscience.

[74]  J. O'Keefe,et al.  The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat. , 1971, Brain research.

[76]  J. O’Keefe,et al.  Boundary Vector Cells in the Subiculum of the Hippocampal Formation , 2009, The Journal of Neuroscience.

[77]  R U Muller,et al.  Head-direction cells recorded from the postsubiculum in freely moving rats. I. Description and quantitative analysis , 1990, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[78]  T. Hafting,et al.  Microstructure of a spatial map in the entorhinal cortex , 2005, Nature.

[79]  Samuel J. Gershman,et al.  Compositional inductive biases in function learning , 2016, Cognitive Psychology.

[80]  R. R. Bush,et al.  A Mathematical Model for Simple Learning , 1951 .

[81]  Y. Niv Learning task-state representations , 2019, Nature Neuroscience.

[82]  M. Shapiro,et al.  A Map for Social Navigation in the Human Brain , 2015, Neuron.

[83]  Thomas T. Hills Animal Foraging and the Evolution of Goal-Directed Cognition , 2006, Cogn. Sci..

[84]  Michael J Kahana,et al.  Hippocampal theta codes for distances in semantic and temporal spaces , 2019, Proceedings of the National Academy of Sciences.

[85]  Michael J Frank,et al.  Within- and across-trial dynamics of human EEG reveal cooperative interplay between reinforcement learning and working memory , 2017, Proceedings of the National Academy of Sciences.

[86]  Marlos C. Machado,et al.  Eigenoption Discovery through the Deep Successor Representation , 2017, ICLR.

[87]  M. Speekenbrink,et al.  It's new, but is it good? How generalization and uncertainty guide the exploration of novel options. , 2020, Journal of experimental psychology. General.

[88]  Joshua T. Abbott,et al.  Random walks on semantic networks can resemble optimal foraging. , 2015, Psychological review.

[89]  Jonathan D. Nelson,et al.  Mapping the unknown: The spatially correlated multi-armed bandit , 2017, bioRxiv.

[90]  M. Speekenbrink,et al.  Putting bandits into context: How function learning supports decision making , 2016, bioRxiv.

[91]  Boris N. Konrad,et al.  Mnemonic Training Reshapes Brain Networks to Support Superior Memory , 2017, Neuron.

[92]  M. Lee,et al.  A Bayesian analysis of human decision-making on bandit problems , 2009 .

[93]  Danielle J. Navarro,et al.  Aversion to Option Loss in a Restless Bandit Task , 2018 .