Rapid learning of predictive maps with STDP and theta phase precession

The predictive map hypothesis is a promising candidate principle for hippocampal function. A favoured formalisation of this hypothesis, called the successor representation, proposes that each place cell encodes the expected state occupancy of its target location in the near future. This predictive framework is supported by behavioural as well as electrophysiological evidence and has desirable consequences for both the generalisability and efficiency of reinforcement learning algorithms. However, it is unclear how the successor representation might be learnt in the brain. Error-driven temporal difference learning, commonly used to learn successor representations in artificial agents, is not known to be implemented in hippocampal networks. Instead, we demonstrate that spike-timing dependent plasticity (STDP), a form of Hebbian learning, acting on temporally compressed trajectories known as “theta sweeps”, is sufficient to rapidly learn a close approximation to the successor representation. The model is biologically plausible – it uses spiking neurons modulated by theta-band oscillations, diffuse and overlapping place cell-like state representations, and experimentally matched parameters. We show how this model maps onto known aspects of hippocampal circuitry and explains substantial variance in the temporal difference successor matrix, consequently giving rise to place cells that demonstrate experimentally observed successor representation-related phenomena including backwards expansion on a 1D track and elongation near walls in 2D. Finally, our model provides insight into the observed topographical ordering of place field sizes along the dorsal-ventral axis by showing this is necessary to prevent the detrimental mixing of larger place fields, which encode longer timescale successor representations, with more fine-grained predictions of spatial location.

[1]  N. Burgess,et al.  Firing rate adaptation affords place cell theta sweeps, phase precession and procession , 2024, bioRxiv.

[2]  Kimberly L. Stachenfeld,et al.  RatInABox: A toolkit for modelling locomotion and neuronal activity in continuous environments , 2023, bioRxiv.

[3]  C. Clopath,et al.  Learning predictive cognitive maps with spiking neurons during behavior and replays , 2023, eLife.

[4]  Emily L. Mackevicius,et al.  Neural learning rules for generating flexible predictions and computing the successor representation , 2022, bioRxiv.

[5]  N. Daw,et al.  Linear reinforcement learning in planning, grid fields, and cognitive control , 2021, Nature Communications.

[6]  C. Barry,et al.  State transitions in the statistically stable place cell population are determined by rate of perceptual change , 2021, bioRxiv.

[7]  M. Tsodyks,et al.  Multiscale representation of very large environments in the hippocampus of flying bats , 2021, Science.

[8]  M. Sheffield,et al.  Distinct place cell dynamics in CA1 and CA3 encode experience in new environments , 2021, Nature Communications.

[9]  C. Barry,et al.  Ripple band phase precession of place cell firing during replay , 2021, Current Biology.

[10]  Neil Burgess,et al.  A general model of hippocampal and dorsal striatal learning and decision making , 2020, Proceedings of the National Academy of Sciences.

[11]  Brad E. Pfeiffer,et al.  Alternating sequences of future and past behavior encoded within hippocampal theta oscillations , 2020, Science.

[12]  Roddy M. Grieves,et al.  Predictive maps in rats and humans for spatial navigation , 2020, Current Biology.

[13]  Eric T. Reifenstein,et al.  Synaptic learning rules for sequence learning , 2020, bioRxiv.

[14]  György Buzsáki,et al.  Cooling of Medial Septum Reveals Theta Phase Lag Coordination of Hippocampal Cell Assemblies , 2019, Neuron.

[15]  Caswell Barry,et al.  Neurobiological successor features for spatial navigation , 2019, bioRxiv.

[16]  Maneesh Sahani,et al.  A neurally plausible model learns successor representations in partially observable environments , 2019, NeurIPS.

[17]  Mattias P. Karlsson,et al.  Constant Sub-second Cycling between Representations of Possible Futures in the Hippocampus , 2019, Cell.

[18]  Janina Ferbinteanu,et al.  Place cell firing cannot support navigation without intact septal circuits , 2018, bioRxiv.

[19]  Marc W. Howard,et al.  Predicting the Future with Multi-scale Successor Representations , 2018, bioRxiv.

[20]  Samuel J Gershman,et al.  The Successor Representation: Its Computational Logic and Neural Substrates , 2018, The Journal of Neuroscience.

[21]  Jeffrey L. Gauthier,et al.  A Dedicated Population for Reward Coding in the Hippocampus , 2018, Neuron.

[22]  Neil C. Rabinowitz,et al.  Vector-based navigation using grid-like representations in artificial agents , 2018, Nature.

[23]  Marcelo G Mattar,et al.  Prioritized memory access explains planning and hippocampal replay , 2017, Nature Neuroscience.

[24]  Katie C. Bittner,et al.  Behavioral time scale synaptic plasticity underlies CA1 place fields , 2017, Science.

[25]  Kimberly L. Stachenfeld,et al.  The hippocampus as a predictive map , 2017, Nature Neuroscience.

[26]  Samuel Gershman,et al.  Predictive representations can link model-based reinforcement learning to model-free mechanisms , 2017, bioRxiv.

[27]  M. Botvinick,et al.  The successor representation in human reinforcement learning , 2016, Nature Human Behaviour.

[28]  Johanni Brea,et al.  Prospective Coding by Spiking Neurons , 2016, PLoS Comput. Biol..

[29]  Neil Burgess,et al.  Using Grid Cells for Navigation , 2015, Neuron.

[30]  K. Jeffery,et al.  Grid Cells Form a Global Representation of Connected Environments , 2015, Current Biology.

[31]  David J. Foster,et al.  Dissociation between the Experience-Dependent Development of Hippocampal Theta Sequences and Single-Trial Phase Precession , 2015, The Journal of Neuroscience.

[32]  T. Zentall When animals misbehave: Analogs of human biases and suboptimal choice , 2015, Behavioural Processes.

[33]  Samuel Gershman,et al.  Design Principles of the Hippocampal Cognitive Map , 2014, NIPS.

[34]  E. Lein,et al.  Functional organization of the hippocampal longitudinal axis , 2014, Nature Reviews Neuroscience.

[35]  Angus Chadwick,et al.  Independent theta phase coding accounts for CA1 population sequences and enables flexible remapping , 2014, bioRxiv.

[36]  C. Barry,et al.  Theta phase precession of grid and place cell firing in open environments , 2014, Philosophical Transactions of the Royal Society B: Biological Sciences.

[37]  Hugo J. Spiers,et al.  Place Field Repetition and Purely Local Remapping in a Multicompartment Environment , 2013, Cerebral cortex.

[38]  Michael X. Cohen,et al.  Frontal theta reflects uncertainty and unexpectedness during exploration and exploitation. , 2012, Cerebral cortex.

[39]  Thomas T. Hills,et al.  Cognitive search : evolution, algorithms, and the brain , 2012 .

[40]  G. Buzsáki,et al.  Traveling Theta Waves along the Entire Septotemporal Axis of the Hippocampus , 2012, Neuron.

[41]  Kamran Diba,et al.  Activity dynamics and behavioral correlates of CA3 and CA1 hippocampal pyramidal neurons , 2012, Hippocampus.

[42]  Michael E. Hasselmo,et al.  Modeling Boundary Vector Cell Firing Given Optic Flow as a Cue , 2012, PLoS Comput. Biol..

[43]  Uğur M Erdem,et al.  A goal‐directed spatial navigation model using forward trajectory planning based on grid cells , 2012, The European journal of neuroscience.

[44]  Nathaniel D. Daw,et al.  Grid Cells, Place Cells, and Geodesic Generalization for Spatial Reinforcement Learning , 2011, PLoS Comput. Biol..

[45]  E. Save,et al.  Local remapping of place cell firing in the Tolman detour task , 2011, The European journal of neuroscience.

[46]  J. O’Neill,et al.  The reorganization and reactivation of hippocampal maps predict spatial memory performance , 2010, Nature Neuroscience.

[47]  Andrew Philippides,et al.  Dual Coding with STDP in a Spiking Recurrent Neural Network Model of the Hippocampus , 2010, PLoS Comput. Biol..

[48]  T. Hafting,et al.  Frequency of gamma oscillations routes flow of information in the hippocampus , 2009, Nature.

[49]  Emanuel Todorov,et al.  Efficient computation of optimal actions , 2009, Proceedings of the National Academy of Sciences.

[50]  Evgueniy V. Lubenov,et al.  Hippocampal theta oscillations are travelling waves , 2009, Nature.

[51]  M. Moser,et al.  Foreword: Special issue on grid cells , 2008, Scholarpedia.

[52]  T. Hafting,et al.  Finite Scale of Spatial Representation in the Hippocampus , 2008, Science.

[53]  L. Frank,et al.  New Experiences Enhance Coordinated Neural Activity in the Hippocampus , 2008, Neuron.

[54]  Adam Johnson,et al.  Neural Ensembles in CA3 Transiently Encode Paths Forward of the Animal at a Decision Point , 2007, The Journal of Neuroscience.

[55]  A. Boddy,et al.  Molecular targeting of retinoic acid metabolism in neuroblastoma: the role of the CYP26 inhibitor R116010 in vitro and in vivo , 2007, British Journal of Cancer.

[56]  Stephen L. Cowen,et al.  Organization of hippocampal cell assemblies based on theta phase precession , 2006, Hippocampus.

[57]  M. Hasselmo,et al.  2005 Special issue: Hippocampal mechanisms for the context-dependent retrieval of episodes , 2005 .

[58]  T. Hafting,et al.  Microstructure of a spatial map in the entorhinal cortex , 2005, Nature.

[59]  K. Harris Neural signatures of cell assembly organization , 2005, Nature Reviews Neuroscience.

[60]  Peter Dayan,et al.  Temporal difference models describe higher-order learning in humans , 2004, Nature.

[61]  John O'Keefe,et al.  Independent rate and temporal coding in hippocampal pyramidal cells , 2003, Nature.

[62]  Michael E. Hasselmo,et al.  Modeling goal-directed spatial navigation in the rat based on physiological data from the hippocampal formation , 2003, Neural Networks.

[63]  M. R. Mehta,et al.  Role of experience and oscillations in transforming a rate code into a temporal code , 2002, Nature.

[64]  Michael E. Hasselmo,et al.  A Proposed Function for Hippocampal Theta Rhythm: Separate Phases of Encoding and Retrieval Enhance Reversal of Prior Learning , 2002, Neural Computation.

[65]  M. Mehta Neuronal Dynamics of Predictive Coding , 2001, The Neuroscientist : a review journal bringing neurobiology, neurology and psychiatry.

[66]  M. Quirk,et al.  Experience-Dependent Asymmetric Shape of Hippocampal Receptive Fields , 2000, Neuron.

[67]  L. F. Abbott,et al.  A Model of Spatial Map Formation in the Hippocampus of the Rat , 1999, Neural Computation.

[68]  G. Bi,et al.  Synaptic Modifications in Cultured Hippocampal Neurons: Dependence on Spike Timing, Synaptic Strength, and Postsynaptic Cell Type , 1998, The Journal of Neuroscience.

[69]  J O'Keefe,et al.  Robotic and neuronal simulation of the hippocampus and rat navigation. , 1997, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[70]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[71]  J. Lisman,et al.  Hippocampal CA3 region predicts memory sequences: accounting for the phase precession of place cells. , 1996, Learning & memory.

[72]  B. McNaughton,et al.  Replay of Neuronal Firing Sequences in Rat Hippocampus During Sleep Following Spatial Experience , 1996, Science.

[73]  F. Motamedi,et al.  Effects of reversible inactivations of the medial septal area on reference and working memory versions of the Morris water maze , 1996, Brain Research.

[74]  B. McNaughton,et al.  Reactivation of hippocampal ensemble memories during sleep. , 1994, Science.

[75]  B L McNaughton,et al.  Dynamics of the hippocampal ensemble code for space. , 1993, Science.

[76]  Peter Dayan,et al.  Improving Generalization for Temporal Difference Learning: The Successor Representation , 1993, Neural Computation.

[77]  J. O’Keefe,et al.  Phase relationship between hippocampal place units and the EEG theta rhythm , 1993, Hippocampus.

[78]  R. Muller,et al.  The hippocampus as a cognitive graph (abridged version) , 1991, Hippocampus.

[79]  R. Muller,et al.  Head-direction cells recorded from the postsubiculum in freely moving rats. II. Effects of environmental manipulations , 1990, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[80]  T. J. Walsh,et al.  Intraseptal administration of muscimol produces dose-dependent memory impairments in the rat. , 1989, Behavioral and neural biology.

[81]  R. Morris,et al.  Place navigation impaired in rats with hippocampal lesions , 1982, Nature.

[82]  M. Eckardt The Hippocampus as a Cognitive Map , 1980 .

[83]  J. O'Keefe,et al.  The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat. , 1971, Brain research.

[84]  W. Scoville,et al.  LOSS OF RECENT MEMORY AFTER BILATERAL HIPPOCAMPAL LESIONS , 1957, Journal of neurology, neurosurgery, and psychiatry.

[85]  E. Tolman Cognitive maps in rats and men. , 1948, Psychological review.

[86]  Tiejun,et al.  Firing rate adaptation in continuous attractor neural networks accounts for theta phase shift of hippocampal place cells , 2022 .

[87]  Stephen W. Carden,et al.  An Introduction to Reinforcement Learning , 2013 .

[88]  N. Daw Model-based reinforcement learning as cognitive search : Neurocomputational theories , 2012 .

[89]  Kenji Doya,et al.  Reinforcement Learning in Continuous Time and Space , 2000, Neural Computation.

[90]  David S. Touretzky,et al.  The Role of the Hippocampus in Solving the Morris Water Maze , 1998, Neural Computation.

[91]  B. McNaughton,et al.  Theta phase precession in hippocampal neuronal populations and the compression of temporal sequences , 1996, Hippocampus.