Replay and compositional computation

1DeepMind, London, UK 2Max Planck UCL Centre for Computational Psychiatry and Ageing Research, London, UK 3Wellcome Centre for Human Neuroimaging, University College London, London, UK 4Wellcome Centre for Integrative Neuroimaging, University of Oxford, Oxford, UK 5Institute of Ophthalmology, University College London, London, UK 6State Key Laboratory of Cognitive Neuroscience and Learning, IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, China 7Chinese Institute for Brain Research, Beijing, China 8Max Planck Institute for Biological Cybernetics, Tubingen, Germany 9University of Tubingen, Tubingen, Germany

[1]  David J. Foster,et al.  Flexible rerouting of hippocampal replay sequences around changing barriers in the absence of global place field remapping , 2022, Neuron.

[2]  M. Lengyel,et al.  Planning in the brain , 2022, Neuron.

[3]  James C. R. Whittington,et al.  Relating transformers to models and neural representations of the hippocampal formation , 2021, ICLR.

[4]  Gergő Orbán,et al.  Sampling motion trajectories during hippocampal theta sequences , 2021, bioRxiv.

[5]  Andrew M. Saxe,et al.  Organizing memories for generalization in complementary learning systems , 2021, bioRxiv.

[6]  Nathan Lepora,et al.  Learning offline: memory replay in biological and artificial reinforcement learning , 2021, Trends in Neurosciences.

[7]  Samuel J. Gershman,et al.  Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning , 2021, ArXiv.

[8]  N. Daw,et al.  Formalizing planning and information search in naturalistic decision-making , 2021, Nature Neuroscience.

[9]  Armando Solar-Lezama,et al.  DreamCoder: bootstrapping inductive program synthesis with wake-sleep library learning , 2021, PLDI.

[10]  Ryan J. Low,et al.  Geometry of abstract learned knowledge in the hippocampus , 2021, Nature.

[11]  Timothy H. Muller,et al.  Generative replay for compositional visual understanding in the prefrontal-hippocampal circuit , 2021, bioRxiv.

[12]  Daniel N. Barry,et al.  A neural network account of memory replay and knowledge consolidation , 2021, bioRxiv.

[13]  David J. Foster,et al.  Prefrontal Cortical Neurons Are Selective for Non-Local Hippocampal Representations during Replay and Behavior , 2021, The Journal of Neuroscience.

[14]  N. Daw,et al.  Experience replay is associated with efficient nonlocal learning , 2021, Science.

[15]  P. Dayan,et al.  Optimism and pessimism in optimised replay , 2021, bioRxiv.

[16]  Lennart Wittkuhn,et al.  Replay in minds and machines , 2021, Neuroscience & Biobehavioral Reviews.

[17]  Kazuki Irie,et al.  Linear Transformers Are Secretly Fast Weight Programmers , 2021, ICML.

[18]  Miriam C. Klein-Flügge,et al.  Activation and disruption of a neural mechanism for novel choice in monkeys , 2020, Nature.

[19]  Brenden M. Lake,et al.  Word meaning in minds and machines , 2020, Psychological review.

[20]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[21]  Daniel N. Barry,et al.  vmPFC Drives Hippocampal Processing during Autobiographical Memory Recall Regardless of Remoteness , 2020, bioRxiv.

[22]  Chen Sun,et al.  Hippocampal neurons represent events as transferable units of experience , 2020, Nature Neuroscience.

[23]  Peter Dayan,et al.  The roles of online and offline replay in planning , 2020, bioRxiv.

[24]  Jessica B. Hamrick,et al.  Combining Q-Learning and Search with Amortized Value Estimates , 2019, ICLR.

[25]  Demis Hassabis,et al.  Mastering Atari, Go, chess and shogi by planning with a learned model , 2019, Nature.

[26]  Caswell Barry,et al.  The Tolman-Eichenbaum Machine: Unifying Space and Relational Memory through Generalization in the Hippocampal Formation , 2019, Cell.

[27]  Mattias P. Karlsson,et al.  Constant Sub-second Cycling between Representations of Possible Futures in the Hippocampus , 2019, Cell.

[28]  Steven M Frankland,et al.  Concepts and Compositionality: In Search of the Brain's Language of Thought. , 2020, Annual review of psychology.

[29]  Wojciech M. Czarnecki,et al.  Grandmaster level in StarCraft II using multi-agent reinforcement learning , 2019, Nature.

[30]  Matthew P. H. Gardner,et al.  Complementary Task Structure Representations in Hippocampus and Orbitofrontal Cortex during an Odor Sequence Task , 2019, Current Biology.

[31]  Anthony M. Zador,et al.  A critique of pure learning and what artificial neural networks can learn from animal brains , 2019, Nature Communications.

[32]  Guillaume Hennequin,et al.  Cortical-like dynamics in recurrent circuits optimized for sampling-based probabilistic inference , 2019, Nature Neuroscience.

[33]  Timothy E. J. Behrens,et al.  Human Replay Spontaneously Reorganizes Experience , 2019, Cell.

[34]  Matteo Hessel,et al.  When to use parametric models in reinforcement learning? , 2019, NeurIPS.

[35]  Chuang Gan,et al.  The Neuro-Symbolic Concept Learner: Interpreting Scenes Words and Sentences from Natural Supervision , 2019, ICLR.

[36]  Jozsef Csicsvari,et al.  Hippocampal Reactivation of Random Trajectories Resembling Brownian Diffusion , 2019, Neuron.

[37]  Tom Eccles,et al.  An investigation of model-free planning , 2019, ICML.

[38]  Jozsef Csicsvari,et al.  Assembly Responses of Hippocampal CA1 Place Cells Predict Learned Behavior in Goal-Directed Spatial Tasks on the Radial Eight-Arm Maze , 2019, Neuron.

[39]  Surya Ganguli,et al.  A mathematical theory of semantic development in deep neural networks , 2018, Proceedings of the National Academy of Sciences.

[40]  G. Buzsáki,et al.  Space and Time: The Hippocampus as a Sequence Generator , 2018, Trends in Cognitive Sciences.

[41]  C. Summerfield,et al.  Neural structure mapping in human probabilistic reward learning , 2018, bioRxiv.

[42]  Razvan Pascanu,et al.  Relational inductive biases, deep learning, and graph networks , 2018, ArXiv.

[43]  Günther Knoblich,et al.  How Memory Replay in Sleep Boosts Creative Problem-Solving , 2018, Trends in Cognitive Sciences.

[44]  Daniel L. K. Yamins,et al.  A Task-Optimized Neural Network Replicates Human Auditory Behavior, Predicts Brain Responses, and Reveals a Cortical Processing Hierarchy , 2018, Neuron.

[45]  Joel Z. Leibo,et al.  Prefrontal cortex as a meta-reinforcement learning system , 2018, bioRxiv.

[46]  Sumit Gulwani,et al.  Neural-Guided Deductive Search for Real-Time Program Synthesis from Examples , 2018, ICLR.

[47]  C. Barry,et al.  The Role of Hippocampal Replay in Memory and Planning , 2018, Current Biology.

[48]  S. Maruyama,et al.  Origins of building blocks of life: A review , 2017, Geoscience Frontiers.

[49]  Marcelo G Mattar,et al.  Prioritized memory access explains planning and hippocampal replay , 2017, Nature Neuroscience.

[50]  C. Barry,et al.  Task Demands Predict a Dynamic Switch in the Content of Awake Hippocampal Replay , 2017, Neuron.

[51]  David J. Foster.  Replay Comes of Age. , 2017, Annual review of neuroscience.

[52]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[53]  Kimberly L. Stachenfeld,et al.  The hippocampus as a predictive map , 2017, Nature Neuroscience.

[54]  Jiwon Kim,et al.  Continual Learning with Deep Generative Replay , 2017, NIPS.

[55]  Dmitriy Aronov,et al.  Mapping of a non-spatial dimension by the hippocampal/entorhinal circuit , 2017, Nature.

[56]  Elad Eban,et al.  A cortical–hippocampal–cortical loop of information processing during memory consolidation , 2016, Nature Neuroscience.

[57]  József Fiser,et al.  Neural Variability and Sampling-Based Probabilistic Representations in the Visual Cortex , 2016, Neuron.

[58]  Zeb Kurth-Nelson,et al.  Fast Sequences of Non-spatial State Representations in Humans , 2016, Neuron.

[59]  Timothy E. J. Behrens,et al.  Organizing conceptual knowledge in humans with a gridlike code , 2016, Science.

[60]  Caswell Barry,et al.  Coordinated grid and place cell replay during rest , 2016, Nature Neuroscience.

[61]  Erin L. Rich,et al.  Decoding subjective decisions from orbitofrontal cortex , 2016, Nature Neuroscience.

[62]  Joshua B. Tenenbaum,et al.  Building machines that learn and think like people , 2016, Behavioral and Brain Sciences.

[63]  H. Eichenbaum,et al.  Complementary Functional Organization of Neuronal Activity Patterns in the Perirhinal, Lateral Entorhinal, and Medial Entorhinal Cortices , 2016, The Journal of Neuroscience.

[64]  N. Fortin,et al.  Nonspatial Sequence Coding in CA1 Neurons , 2016, The Journal of Neuroscience.

[65]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[66]  Delia Silva,et al.  Trajectory events across hippocampal place-cells require previous experience , 2015, Nature Neuroscience.

[67]  G. Buzsáki.  Hippocampal sharp wave‐ripple: A cognitive biomarker for episodic memory and planning , 2015, Hippocampus.

[68]  Brad E. Pfeiffer,et al.  Autoassociative dynamics in the generation of sequences of hippocampal place cells , 2015, Science.

[69]  M. Shapiro,et al.  A Map for Social Navigation in the Human Brain , 2015, Neuron.

[70]  Matthew T. Kaufman,et al.  A neural network that finds a naturalistic solution for the production of muscle activity , 2015, Nature Neuroscience.

[71]  D. Hassabis,et al.  Hippocampal place cells construct reward related sequences through unexplored space , 2015, eLife.

[72]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[73]  Andrew M. Wikenheiser,et al.  Hippocampal theta sequences reflect current goals , 2015, Nature Neuroscience.

[74]  Thomas T. Hills,et al.  Exploration versus exploitation in space, mind, and society , 2015, Trends in Cognitive Sciences.

[75]  Roddy M. Grieves,et al.  The Neural Basis of a Cognitive Map , 2015 .

[76]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[77]  Matthijs A. A. van der Meer,et al.  Internally generated sequences in learning and executing goal-directed behavior , 2014, Trends in Cognitive Sciences.

[78]  P. Dayan,et al.  The algorithmic anatomy of model-based evaluation , 2014, Philosophical Transactions of the Royal Society B: Biological Sciences.

[79]  H. Eichenbaum.  Time cells in the hippocampus: a new dimension for mapping memories , 2014, Nature Reviews Neuroscience.

[80]  Michael D. Howard,et al.  Complementary Learning Systems , 2014, Cogn. Sci..

[81]  Ha Hong,et al.  Performance-optimized hierarchical models predict neural responses in higher visual cortex , 2014, Proceedings of the National Academy of Sciences.

[82]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[83]  W. Newsome,et al.  Context-dependent computation by recurrent dynamics in prefrontal cortex , 2013, Nature.

[84]  J. Born,et al.  Sleep for preserving and transforming episodic memory. , 2013, Annual review of neuroscience.

[85]  Brad E. Pfeiffer,et al.  Hippocampal place cell sequences depict future paths to remembered goals , 2013, Nature.

[86]  J. Lisman,et al.  The Theta-Gamma Neural Code , 2013, Neuron.

[87]  Margaret F. Carr,et al.  Hippocampal SWR Activity Predicts Correct Decisions during the Initial Learning of an Alternation Task , 2013, Neuron.

[88]  Nathaniel J. Killian,et al.  A map of visual space in the primate entorhinal cortex , 2012, Nature.

[89]  Daniel Bendor,et al.  Biasing the content of hippocampal replay during sleep , 2012, Nature Neuroscience.

[90]  Rens Bod,et al.  Structured cognition and neural systems: From rats to language , 2012, Neuroscience & Biobehavioral Reviews.

[91]  Adam N. Sanborn,et al.  Bridging Levels of Analysis for Probabilistic Models of Cognition , 2012 .

[92]  Y. Dudai.  The restless engram: consolidations never end. , 2012, Annual review of neuroscience.

[93]  L. Frank,et al.  Awake Hippocampal Sharp-Wave Ripples Support Spatial Memory , 2012, Science.

[94]  Margaret L. Schlichting,et al.  The hippocampus and inferential reasoning: building memories to navigate future decisions , 2012, Front. Hum. Neurosci..

[95]  Wolfgang Maass,et al.  Neural Dynamics as Sampling: A Model for Stochastic Computation in Recurrent Networks of Spiking Neurons , 2011, PLoS Comput. Biol..

[96]  C. Stark,et al.  Pattern separation in the hippocampus , 2011, Trends in Neurosciences.

[97]  Cyriel M. A. Pennartz,et al.  The Construction of Semantic Memory: Grammar-Based Representations Learned from Relational Episodic Information , 2011, Front. Comput. Neurosci..

[98]  G. Dragoi,et al.  Preplay of future place cell sequences by hippocampal cellular assemblies , 2011, Nature.

[99]  József Fiser,et al.  Spontaneous Cortical Activity Reveals Hallmarks of an Optimal Internal Model of the Environment , 2011, Science.

[100]  György Buzsáki,et al.  Neural Syntax: Cell Assemblies, Synapsembles, and Readers , 2010, Neuron.

[101]  J. Tenenbaum,et al.  Probabilistic models of cognition: exploring representations and inductive biases , 2010, Trends in Cognitive Sciences.

[102]  James L. McClelland,et al.  Letting structure emerge: connectionist and dynamical systems approaches to cognition , 2010, Trends in Cognitive Sciences.

[103]  Matthijs A. A. van der Meer,et al.  Hippocampal Replay Is Not a Simple Function of Experience , 2010, Neuron.

[104]  R. Buckner.  The role of the hippocampus in prediction and imagination. , 2010, Annual review of psychology.

[105]  G. Buzsáki,et al.  Selective suppression of hippocampal ripples impairs spatial memory , 2009, Nature Neuroscience.

[106]  Matthijs A. A. van der Meer,et al.  Frontiers in Integrative Neuroscience Integrative Neuroscience Covert Expectation-of-reward in Rat Ventral Striatum at Decision Points , 2022 .

[107]  Andre Cohen,et al.  An object-oriented representation for efficient reinforcement learning , 2008, ICML '08.

[108]  J. W. Rudy,et al.  The hippocampal indexing theory and episodic memory: Updating the index , 2007, Hippocampus.

[109]  Matthew Botvinick,et al.  From Numerosity to Ordinal Rank: A Gain-Field Model of Serial Order Representation in Cortical Working Memory , 2007, The Journal of Neuroscience.

[110]  Dorothy Tse,et al.  References and Notes Supporting Online Material Materials and Methods Figs. S1 to S5 Tables S1 to S3 Electron Impact (ei) Mass Spectra Chemical Ionization (ci) Mass Spectra References Schemas and Memory Consolidation Research Articles Research Articles Research Articles Research Articles , 2022 .

[111]  A. Treves,et al.  Hippocampal remapping and grid realignment in entorhinal cortex , 2007, Nature.

[112]  M. Moser,et al.  Pattern Separation in the Dentate Gyrus and CA3 of the Hippocampus , 2007, Science.

[113]  D. Hassabis,et al.  Patients with hippocampal amnesia cannot imagine new experiences , 2007, Proceedings of the National Academy of Sciences.

[114]  S. Harnad.  Symbol grounding problem , 1990, Scholarpedia.

[115]  M. Wilson,et al.  Coordinated memory replay in the visual cortex and hippocampus during sleep , 2007, Nature Neuroscience.

[116]  H. Eichenbaum,et al.  Evolution of declarative memory , 2006, Hippocampus.

[117]  B. Hars,et al.  In memory of consolidation. , 2006, Learning & memory.

[118]  J. Knierim,et al.  Hippocampal place cells: Parallel input streams, subregional processing, and implications for episodic memory , 2006, Hippocampus.

[119]  Bruce L. McNaughton,et al.  Path integration and the neural basis of the 'cognitive map' , 2006, Nature Reviews Neuroscience.

[120]  F. van der Velde,et al.  Neural blackboard architectures of combinatorial structures in cognition , 2006, Behavioral and Brain Sciences.

[121]  Michael J. Frank,et al.  Making Working Memory Work: A Computational Model of Learning in the Prefrontal Cortex and Basal Ganglia , 2006, Neural Computation.

[122]  A. David Redish,et al.  Hippocampal replay contributes to within session learning in a temporal difference reinforcement learning model , 2005, Neural Networks.

[123]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[124]  P. Dayan,et al.  Off-line replay maintains declarative memories in a model of hippocampal-neocortical interactions , 2004, Nature Neuroscience.

[125]  J. Csicsvari,et al.  Organization of cell assemblies in the hippocampus , 2003, Nature.

[126]  K. Holyoak,et al.  A symbolic-connectionist theory of relational inference and generalization. , 2003, Psychological review.

[127]  R. O’Reilly,et al.  Conjunctive representations in learning and memory: principles of cortical and hippocampal function. , 2001, Psychological review.

[128]  W. Singer,et al.  Temporal binding and the neural correlates of sensory awareness , 2001, Trends in Cognitive Sciences.

[129]  M. Wilson,et al.  Temporally Structured Replay of Awake Hippocampal Ensemble Activity during Rapid Eye Movement Sleep , 2001, Neuron.

[130]  J. Csicsvari,et al.  Replay and Time Compression of Recurring Spike Sequences in the Hippocampus , 1999, The Journal of Neuroscience.

[131]  James L. McClelland,et al.  Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. , 1995, Psychological review.

[132]  L. Squire,et al.  Retrograde amnesia and memory consolidation: a neurobiological perspective , 1995, Current Opinion in Neurobiology.

[133]  J E Lisman,et al.  Storage of 7 +/- 2 short-term memories in oscillatory subcycles , 1995, Science.

[134]  J. Hodges.  Memory, Amnesia and the Hippocampal System , 1995 .

[135]  B. McNaughton,et al.  Reactivation of hippocampal ensemble memories during sleep. , 1994, Science.

[136]  E. Rolls,et al.  Computational analysis of the role of the hippocampus in memory , 1994, Hippocampus.

[137]  M. Moscovitch.  Memory and Working-with-Memory: A Component Process Model Based on Modules and Central Systems , 1992, Journal of Cognitive Neuroscience.

[138]  I. Biederman,et al.  Dynamic binding in a neural network for shape recognition. , 1992, Psychological review.

[139]  M. Goodale,et al.  Separate visual pathways for perception and action , 1992, Trends in Neurosciences.

[140]  Long Ji Lin,et al.  Programming Robots Using Reinforcement Learning and Teaching , 1991, AAAI.

[141]  Richard S. Sutton,et al.  Dyna, an integrated architecture for learning, planning, and reacting , 1990, SGAR.

[142]  Geoffrey E. Hinton.  Tensor Product Variable Binding and the Representation of Symbolic Structures in Connectionist Systems , 1991 .

[143]  G. Buzsáki.  Two-stage model of memory trace formation: A role for “noisy” brain states , 1989, Neuroscience.

[144]  Allen Newell,et al.  Elements of a theory of human problem solving. , 1958 .