A model for navigation in unknown environments based on a reservoir of hippocampal sequences

Hippocampal place cell populations are activated in sequences on multiple time scales during active behavior, resting and sleep states, suggesting that these sequences are the genuine dynamical motifs of the hippocampal circuit. Recently, prewired hippocampal place cell sequences have even been reported to correlate to future behaviors, but so far there is no explanation of what could be the computational benefits of such a mapping between intrinsic dynamical structure and external sensory inputs. Here, I propose a computational model in which a set of predefined internal sequences is used as a dynamical reservoir to construct a spatial map of a large unknown maze based on only a small number of salient landmarks. The model is based on a new variant of temporal difference learning and implements a simultaneous localization and mapping algorithm. As a result sequences during intermittent replay periods can be decoded as spatial trajectories and improve navigation performance, which supports the functional interpretation of replay to consolidate memories of motor actions.

[1]  H. C. LONGUET-HIGGINS,et al.  Non-Holographic Associative Memory , 1969, Nature.

[2]  D. Ji,et al.  Hippocampal awake replay in fear memory retrieval , 2017, Nature Neuroscience.

[3]  Lisa M. Giocomo,et al.  Phase precession and variable spatial scaling in a periodic attractor map model of medial entorhinal grid cells with realistic after‐spike dynamics , 2012, Hippocampus.

[4]  G. Dragoi,et al.  Preconfigured patterns are the primary driver of offline multi‐neuronal sequence replay , 2018, Hippocampus.

[5]  J. B. Ranck,et al.  Spatial firing patterns of hippocampal complex-spike cells in a fixed environment , 1987, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[6]  J. O’Neill,et al.  Place-Selective Firing of CA1 Pyramidal Cells during Sharp Wave/Ripple Network Patterns in Exploratory Behavior , 2006, Neuron.

[7]  Rosa Cossart,et al.  Awake hippocampal reactivations project onto orthogonal neuronal assemblies , 2016, Science.

[8]  G L Shulman,et al.  INAUGURAL ARTICLE by a Recently Elected Academy Member:A default mode of brain function , 2001 .

[9]  Moshe Abeles,et al.  On Embedding Synfire Chains in a Balanced Network , 2003, Neural Computation.

[10]  Andres D. Grosmark,et al.  Diversity in neural firing dynamics supports both rigid and learned hippocampal sequences , 2016, Science.

[11]  Jeffrey L. Gauthier,et al.  A Dedicated Population for Reward Coding in the Hippocampus , 2018, Neuron.

[12]  Matthijs A. A. van der Meer,et al.  Hippocampal Replay Is Not a Simple Function of Experience , 2010, Neuron.

[13]  Hong Zhang,et al.  Fast-SeqSLAM: A fast appearance based place recognition algorithm , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[14]  L. Abbott,et al.  Cascade Models of Synaptically Stored Memories , 2005, Neuron.

[15]  M Tsodyks,et al.  Attractor neural network models of spatial maps in hippocampus , 1999, Hippocampus.

[16]  D. Touretzky,et al.  Cognitive maps beyond the hippocampus , 1997, Hippocampus.

[17]  Mehmet Fatih Yanik,et al.  A Hippocampal Model for Behavioral Time Acquisition and Fast Bidirectional Replay of Spatio-Temporal Memory Sequences , 2018, bioRxiv.

[18]  Kimberly L. Stachenfeld,et al.  The hippocampus as a predictive map , 2017, Nature Neuroscience.

[19]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[20]  Katsunori Kitano,et al.  spike-timing-dependent plasticity , 2002 .

[21]  D S Touretzky,et al.  Theory of rodent navigation based on interacting representations of space , 1996, Hippocampus.

[22]  A. Redish.  Beyond the Cognitive Map: From Place Cells to Episodic Memory , 1999 .

[23]  Frances S. Chance,et al.  Hippocampal Phase Precession from Dual Input Components , 2012, The Journal of Neuroscience.

[24]  Thomas J. Wills,et al.  Theta-Modulated Place-by-Direction Cells in the Hippocampal Formation in the Rat , 2004, The Journal of Neuroscience.

[25]  Daniel Medina,et al.  Inhomogeneous Sparseness Leads to Dynamic Instability During Sequence Memory Recall in a Recurrent Neural Network Model , 2013, Journal of mathematical neuroscience.

[26]  György Buzsáki,et al.  Hippocampal place cell assemblies are speed-controlled oscillators , 2007, Proceedings of the National Academy of Sciences.

[27]  Mayank R. Mehta,et al.  Causal Influence of Visual Cues on Hippocampal Directional Selectivity , 2016, Cell.

[28]  Anoopum S. Gupta,et al.  Segmentation of spatial experience by hippocampal theta sequences , 2012, Nature Neuroscience.

[29]  Richard Kempter,et al.  Memory Capacity for Sequences in a Recurrent Network with Biological Constraints , 2006 .

[30]  Neil Burgess,et al.  Models of place and grid cell firing and theta rhythmicity , 2011, Current Opinion in Neurobiology.

[31]  Stefan Glasauer,et al.  Idiothetic navigation in Gerbils and Humans , 1991 .

[32]  Bruce L. McNaughton,et al.  An Information-Theoretic Approach to Deciphering the Hippocampal Code , 1992, NIPS.

[33]  R. Legenstein,,et al.  Input Prediction and Autonomous Movement Analysis in Recurrent Circuits of Spikitfg Neurons , 2003, Reviews in the neurosciences.

[34]  John J. Hopfield,et al.  Rapid, parallel path planning by propagating wavefronts of spiking neural activity , 2012, Front. Comput. Neurosci..

[35]  David J. Foster,et al.  Hippocampal theta sequences , 2007, Hippocampus.

[36]  Siavash Ahmadi,et al.  DENTATE NETWORK ACTIVITY IS NECESSARY FOR SPATIAL WORKING MEMORY BY SUPPORTING CA3 SHARP-WAVE RIPPLE GENERATION AND PROSPECTIVE FIRING OF CA3 NEURONS , 2017, Nature Neuroscience.

[37]  Richard S. Sutton,et al.  Learning to predict by the methods of temporal differences , 1988, Machine Learning.

[38]  Matthew A. Wilson,et al.  Hippocampal Replay of Extended Experience , 2009, Neuron.

[39]  R. Kempter,et al.  Synaptic tagging, evaluation of memories, and the distal reward problem. , 2010, Learning & memory.

[40]  Albert K. Lee,et al.  Memory of Sequential Experience in the Hippocampus during Slow Wave Sleep , 2002, Neuron.

[41]  Y. Benjamini,et al.  Quantifying the buildup in extent and complexity of free exploration in mice , 2011, Proceedings of the National Academy of Sciences.

[42]  E T Rolls,et al.  Computational constraints suggest the need for two distinct input systems to the hippocampal CA3 network , 1992, Hippocampus.

[43]  A. Treves,et al.  Distinct Ensemble Codes in Hippocampal Areas CA3 and CA1 , 2004, Science.

[44]  Olivia V Haas,et al.  The hippocampal code for space in Mongolian gerbils , 2019, Hippocampus.

[45]  Aude Oliva,et al.  Visual long-term memory has a massive storage capacity for object details , 2008, Proceedings of the National Academy of Sciences.

[46]  Wulfram Gerstner,et al.  Reinforcement Learning Using a Continuous Time Actor-Critic Framework with Spiking Neurons , 2013, PLoS Comput. Biol..

[47]  L. Frank,et al.  Rewarded Outcomes Enhance Reactivation of Experience in the Hippocampus , 2009, Neuron.

[48]  George Dragoi,et al.  Distinct preplay of multiple novel spatial experiences in the rat , 2013, Proceedings of the National Academy of Sciences.

[49]  T. Hafting,et al.  Frequency of gamma oscillations routes flow of information in the hippocampus , 2009, Nature.

[50]  G. Buzsáki,et al.  Long-duration hippocampal sharp wave ripples improve memory , 2019, Science.

[51]  Christian Leibold,et al.  Modality‐specific Subpopulations of Place Fields Coexist in the Hippocampus , 2019, Cerebral cortex.

[52]  H. Eichenbaum,et al.  Hippocampal Neurons Encode Information about Different Types of Memory Episodes Occurring in the Same Location , 2000, Neuron.

[53]  W. Schultz,et al.  Learning of sequential movements by neural network model with dopamine-like reinforcement signal , 1998, Experimental Brain Research.

[54]  R. Kempter,et al.  Sparseness constrains the prolongation of memory lifetime via synaptic metaplasticity. , 2008, Cerebral cortex.

[55]  Matthijs A. A. van der Meer,et al.  Frontiers in Integrative Neuroscience Integrative Neuroscience Covert Expectation-of-reward in Rat Ventral Striatum at Decision Points , 2022 .

[56]  B L McNaughton,et al.  Path Integration and Cognitive Mapping in a Continuous Attractor Neural Network Model , 1997, The Journal of Neuroscience.

[57]  Mandyam V. Srinivasan,et al.  Where paths meet and cross: navigation by path integration in the desert ant and the honeybee , 2015, Journal of Comparative Physiology A.

[58]  J. Csicsvari,et al.  Replay and Time Compression of Recurring Spike Sequences in the Hippocampus , 1999, The Journal of Neuroscience.

[59]  Gordon Wyeth,et al.  SeqSLAM: Visual route-based navigation for sunny summer days and stormy winter nights , 2012, 2012 IEEE International Conference on Robotics and Automation.

[60]  David J. Foster,et al.  A model of hippocampally dependent navigation, using the temporal difference learning rule , 2000, Hippocampus.

[61]  A. Grinvald,et al.  Spontaneously emerging cortical representations of visual attributes , 2003, Nature.

[62]  D. Eilam,et al.  Home base behavior of rats (Rattus norvegicus) exploring a novel environment , 1989, Behavioural Brain Research.

[63]  David J. Foster.  Replay Comes of Age. , 2017, Annual review of neuroscience.

[64]  Matthijs A. A. van der Meer,et al.  Integrating hippocampus and striatum in decision-making , 2007, Current Opinion in Neurobiology.

[65]  Asohan Amarasingham,et al.  Internally Generated Cell Assembly Sequences in the Rat Hippocampus , 2008, Science.

[66]  J. O’Keefe,et al.  Geometric determinants of the place fields of hippocampal neurons , 1996, Nature.

[67]  Matthijs A. A. van der Meer,et al.  Exploring the role of context‐dependent hippocampal activity in spatial alternation behavior , 2007, Hippocampus.

[68]  Stefano Fusi,et al.  Long Memory Lifetimes Require Complex Synapses and Limited Sparseness , 2007, Frontiers Comput. Neurosci..

[69]  Johannes Schemmel,et al.  Reward-based learning under hardware constraints—using a RISC processor embedded in a neuromorphic substrate , 2013, Front. Neurosci..

[70]  Michael E. Hasselmo,et al.  A hierarchical model of goal directed navigation selects trajectories in a visual environment , 2015, Neurobiology of Learning and Memory.

[71]  J. O'Keefe,et al.  The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat. , 1971, Brain research.

[72]  Andrew M. Wikenheiser,et al.  Hippocampal theta sequences reflect current goals , 2015, Nature Neuroscience.

[73]  Magdalene I. Schlesiger,et al.  Hippocampal CA1 replay becomes less prominent but more rigid without inputs from medial entorhinal cortex , 2019, Nature Communications.

[74]  Jadin C. Jackson,et al.  Hippocampal Sharp Waves and Reactivation during Awake States Depend on Repeated Sequential Experience , 2006, The Journal of Neuroscience.

[75]  H. T. Blair,et al.  Cosine Directional Tuning of Theta Cell Burst Frequencies: Evidence for Spatial Coding by Oscillatory Interference , 2011, The Journal of Neuroscience.

[76]  A. Redish,et al.  Disrupting the medial prefrontal cortex alters hippocampal sequences during deliberative decision making. , 2019, Journal of neurophysiology.

[77]  Vijay Balasubramanian,et al.  Environmental deformations dynamically shift the grid cell spatial metric , 2017, bioRxiv.

[78]  L. F. Abbott,et al.  Generating Coherent Patterns of Activity from Chaotic Neural Networks , 2009, Neuron.

[79]  Shantanu P. Jadhav,et al.  Interplay between Hippocampal Sharp-Wave-Ripple Events and Vicarious Trial and Error Behaviors in Decision Making , 2016, Neuron.

[80]  Henry Markram,et al.  Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations , 2002, Neural Computation.

[81]  J. Csicsvari,et al.  Theta phase–specific codes for two-dimensional position, trajectory and heading in the hippocampus , 2008, Nature Neuroscience.

[82]  Brad E. Pfeiffer,et al.  Hippocampal place cell sequences depict future paths to remembered goals , 2013, Nature.

[83]  Dmitriy Aronov,et al.  Mapping of a non-spatial dimension by the hippocampal/entorhinal circuit , 2017, Nature.

[84]  G. Dragoi,et al.  Preplay of future place cell sequences by hippocampal cellular assemblies , 2011, Nature.

[85]  Kefei Liu,et al.  Generative Predictive Codes by Multiplexed Hippocampal Neuronal Tuplets , 2018, Neuron.

[86]  Michael Hasselmo,et al.  GABAergic contributions to gating, timing, and phase precession of hippocampal neuronal activity during theta oscillations , 2012, Hippocampus.

[87]  Jonathan D. Ericson,et al.  Wormholes in virtual space: From cognitive maps to cognitive graphs , 2017, Cognition.

[88]  K M Gothard,et al.  Dynamics of Mismatch Correction in the Hippocampal Ensemble Code for Space: Interaction between Path Integration and Environmental Cues , 1996, The Journal of Neuroscience.

[89]  Peter Dayan,et al.  Improving Generalization for Temporal Difference Learning: The Successor Representation , 1993, Neural Computation.

[90]  Richard S. Sutton,et al.  Time-Derivative Models of Pavlovian Reinforcement , 1990 .

[91]  Daniel Medina,et al.  Re-encoding of associations by recurrent plasticity increases memory capacity , 2014, Front. Synaptic Neurosci..

[92]  Alessandro Treves,et al.  Stable and Rapid Recurrent Processing in Realistic Autoassociative Memories , 1998, Neural Computation.

[93]  Olivier Stasse,et al.  MonoSLAM: Real-Time Single Camera SLAM , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[94]  Matthias Kaschube,et al.  Distributed network interactions and their emergence in developing neocortex , 2018, Nature Neuroscience.

[95]  David J. Foster,et al.  Dissociation between the Experience-Dependent Development of Hippocampal Theta Sequences and Single-Trial Phase Precession , 2015, The Journal of Neuroscience.

[96]  H. Eichenbaum,et al.  The Hippocampus and Disambiguation of Overlapping Sequences , 2002, The Journal of Neuroscience.

[97]  Mark C. Fuhs,et al.  A Spin Glass Model of Path Integration in Rat Medial Entorhinal Cortex , 2006, The Journal of Neuroscience.

[98]  Vassilis Cutsuridis,et al.  Transition between encoding and consolidation/replay dynamics via cholinergic modulation of CAN current: A modeling study , 2015, Hippocampus.

[99]  G. Buzsáki,et al.  Forward and reverse hippocampal place-cell sequences during ripples , 2007, Nature Neuroscience.

[100]  Wilten Nicola,et al.  A diversity of interneurons and Hebbian plasticity facilitate rapid compressible learning in the hippocampus , 2019, Nature Neuroscience.