Predictive Representations in Hippocampal and Prefrontal Hierarchies

As we navigate the world, we use learned representations of relational structures to explore and reach goals. Studies of how relational knowledge enables inference and planning are typically conducted in controlled small-scale settings. It remains unclear, however, how people use stored knowledge in continuously unfolding navigation, e.g., walking long distances in a city. We hypothesized that predictive representations, organized at multiple scales along posterior-anterior prefrontal and hippocampal hierarchies, guide naturalistic navigation. We conducted model-based representational similarity analyses of neuroimaging data measured during navigation of realistically long paths in virtual reality. We tested the pattern similarity of each point–along each path–to a weighted sum of its successor points within different predictive horizons. We found that anterior PFC showed the largest predictive horizons, posterior hippocampus the smallest, with the anterior hippocampus and orbitofrontal regions in between. Our findings offer novel insights into how cognitive maps support hierarchical planning at multiple scales.

[1]  Peter Dayan,et al.  Improving Generalization for Temporal Difference Learning: The Successor Representation , 1993, Neural Computation.

[2]  Blake S. Porter,et al.  Hippocampal Representation of Related and Opposing Memories Develop within Distinct, Hierarchically Organized Neural Schemas , 2014, Neuron.

[3]  P. Janak,et al.  The Orbitofrontal Cortex as Part of a Hierarchical Neural System Mediating Choice between Two Good Options , 2013, The Journal of Neuroscience.

[4]  E. Koechlin,et al.  Anterior Prefrontal Function and the Limits of Human Decision-Making , 2007, Science.

[5]  M. Botvinick,et al.  Neural representations of events arise from temporal community structure , 2013, Nature Neuroscience.

[6]  Marc W. Howard,et al.  Predicting the Future with Multi-scale Successor Representations , 2018, bioRxiv.

[7]  Asaf Gilboa,et al.  A boost of confidence: The role of the ventromedial prefrontal cortex in memory, decision-making, and schemas , 2016, Neuropsychologia.

[8]  E. Koechlin,et al.  The Architecture of Cognitive Control in the Human Prefrontal Cortex , 2003, Science.

[9]  E. Rolls,et al.  The functional neuroanatomy of the human orbitofrontal cortex: evidence from neuroimaging and neuropsychology , 2004, Progress in Neurobiology.

[10]  Christian F. Doeller,et al.  Memory hierarchies map onto the hippocampal long axis in humans , 2015, Nature Neuroscience.

[11]  Brad E. Pfeiffer,et al.  Reverse Replay of Hippocampal Place Cells Is Uniquely Modulated by Changing Reward , 2016, Neuron.

[12]  Zeb Kurth-Nelson,et al.  What Is a Cognitive Map? Organizing Knowledge for Flexible Behavior , 2018, Neuron.

[13]  Ida Momennejad,et al.  Offline replay supports planning in human reinforcement learning , 2018, eLife.

[14]  Morris Moscovitch,et al.  Multiple Scales of Representation along the Hippocampal Anteroposterior Axis in Humans , 2018, Current Biology.

[15]  S. Gilbert Decoding the Content of Delayed Intentions , 2011, The Journal of Neuroscience.

[16]  Raymond J Dolan,et al.  A map of abstract relational knowledge in the human hippocampal–entorhinal cortex , 2017, eLife.

[17]  Hugo J. Spiers,et al.  Solving the detour problem in navigation: a model of prefrontal and hippocampal interactions , 2015, Front. Hum. Neurosci..

[18]  Robert C. Wilson,et al.  Orbitofrontal Cortex as a Cognitive Map of Task Space , 2014, Neuron.

[19]  E. Lein,et al.  Functional organization of the hippocampal longitudinal axis , 2014, Nature Reviews Neuroscience.

[20]  Kevin J. Miller,et al.  Dorsal hippocampus contributes to model-based planning , 2017, Nature Neuroscience.

[21]  D. Bates,et al.  Fitting Linear Mixed-Effects Models Using lme4 , 2014, 1406.5823.

[22]  Nicolas W. Schuck,et al.  Human Orbitofrontal Cortex Represents a Cognitive Map of State Space , 2016, Neuron.

[23]  R. Knight,et al.  The Hippocampus and Entorhinal Cortex Encode the Path and Euclidean Distances to Goals during Navigation , 2014, Current Biology.

[24]  Sanford Weisberg,et al.  An R Companion to Applied Regression , 2010 .

[25]  Alison R Preston,et al.  Ventromedial Prefrontal Cortex Is Necessary for Normal Associative Inference and Memory Integration , 2018, The Journal of Neuroscience.

[26]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[27]  C. Buechel,et al.  Learning of distant state predictions by the orbitofrontal cortex in humans , 2018, bioRxiv.

[28]  E. Tolman Cognitive maps in rats and men. , 1948, Psychological review.

[29]  A. Owen,et al.  Anterior prefrontal cortex: insights into function from anatomy and neuroimaging , 2004, Nature Reviews Neuroscience.

[30]  B. Sievers,et al.  SAMPL: The Spreading Activation and Memory PLasticity Model , 2019, bioRxiv.

[31]  David Badre,et al.  Functional Magnetic Resonance Imaging Evidence for a Hierarchical Organization of the Prefrontal Cortex , 2007, Journal of Cognitive Neuroscience.

[32]  P W Burgess,et al.  Strategy application disorder: the role of the frontal lobes in human multitasking , 2000, Psychological research.

[33]  N. Daw,et al.  Hippocampal Contributions to Model-Based Planning and Spatial Memory , 2019, Neuron.

[34]  Dylan A. Simon,et al.  Neural Correlates of Forward Planning in a Spatial Decision Task in Humans , 2011, The Journal of Neuroscience.

[35]  M. Botvinick,et al.  Statistical learning of temporal community structure in the hippocampus , 2016, Hippocampus.

[36]  Victoria J. H. Ritvo,et al.  Nonmonotonic Plasticity: How Memory Retrieval Drives Learning , 2019, Trends in Cognitive Sciences.

[37]  K. Christoff,et al.  Prefrontal organization of cognitive control according to levels of abstraction , 2009, Brain Research.

[38]  Troy A. Smith,et al.  Human hippocampus represents space and time during retrieval of real-world memories , 2015, Proceedings of the National Academy of Sciences.

[39]  J. Grafman,et al.  Human prefrontal cortex: processing and representational perspectives , 2003, Nature Reviews Neuroscience.

[40]  Hallvard Røe Evensmoen,et al.  Long-axis specialization of the human hippocampus , 2013, Trends in Cognitive Sciences.

[41]  L. Davachi,et al.  Time-resolved neural reinstatement and pattern separation during memory decisions in human hippocampus , 2018, Proceedings of the National Academy of Sciences of the United States of America.

[42]  C. Honey,et al.  Processing Timescales as an Organizing Principle for Primate Cortex , 2015, Neuron.

[43]  T. Hafting,et al.  Finite Scale of Spatial Representation in the Hippocampus , 2008, Science.

[44]  P. Burgess,et al.  The role of rostral prefrontal cortex in prospective memory: A voxel-based lesion study , 2011, Neuropsychologia.

[45]  E. Miller,et al.  Neural circuits subserving the retrieval and maintenance of abstract rules. , 2003, Journal of neurophysiology.

[46]  P. Dayan,et al.  The algorithmic anatomy of model-based evaluation , 2014, Philosophical Transactions of the Royal Society B: Biological Sciences.

[47]  Dagmar Zeithamova,et al.  Hippocampal and Ventral Medial Prefrontal Activation during Retrieval-Mediated Learning Supports Novel Inference , 2012, Neuron.

[48]  C. Grady,et al.  Age differences in the intrinsic functional connectivity of default network subsystems , 2013, Front. Aging Neurosci..

[49]  R W Cox,et al.  AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. , 1996, Computers and biomedical research, an international journal.

[50]  R. Henson,et al.  Differential roles for medial prefrontal and medial temporal cortices in schema-dependent encoding: From congruent to incongruent , 2013, Neuropsychologia.

[51]  Ida Momennejad Learning Structures: Predictive Representations, Replay, and Generalization , 2020, Current Opinion in Behavioral Sciences.

[52]  Lila Davachi,et al.  Similarity Breeds Proximity: Pattern Similarity within and across Contexts Is Related to Later Mnemonic Judgments of Temporal Proximity , 2014, Neuron.

[53]  Abraham Z. Snyder,et al.  Spurious but systematic correlations in functional connectivity MRI networks arise from subject motion , 2012, NeuroImage.

[54]  J. Haynes Brain Reading: Decoding Mental States From Brain Activity In Humans , 2011 .

[55]  Andrew M. Wikenheiser,et al.  Over the river, through the woods: cognitive maps in the hippocampus and orbitofrontal cortex , 2016, Nature Reviews Neuroscience.

[56]  J. Fox Effect Displays in R for Generalised Linear Models , 2003 .

[57]  Timothy E. J. Behrens,et al.  Organizing conceptual knowledge in humans with a gridlike code , 2016, Science.

[58]  Timothy Edward John Behrens,et al.  How Green Is the Grass on the Other Side? Frontopolar Cortex and the Evidence in Favor of Alternative Courses of Action , 2009, Neuron.

[59]  I. Momennejad,et al.  Encoding of Prospective Tasks in the Human Prefrontal Cortex under Varying Task Loads , 2013, The Journal of Neuroscience.

[60]  B. McNaughton,et al.  Comparison of spatial firing characteristics of units in dorsal and ventral hippocampus of the rat , 1994, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[61]  P. Dayan,et al.  Model-based influences on humans’ choices and striatal prediction errors , 2011, Neuron.

[62]  Caswell Barry,et al.  The Tolman-Eichenbaum Machine: Unifying Space and Relational Memory through Generalization in the Hippocampal Formation , 2019, Cell.

[63]  Danesh Shahnazian,et al.  Subgoal- and Goal-related Reward Prediction Errors in Medial Prefrontal Cortex , 2019, Journal of Cognitive Neuroscience.

[64]  Margaret L. Schlichting,et al.  Hippocampal representations as a function of time, subregion, and brain state , 2018, Neurobiology of Learning and Memory.

[65]  R. Passingham The hippocampus as a cognitive map J. O'Keefe & L. Nadel, Oxford University Press, Oxford (1978). 570 pp., £25.00 , 1979, Neuroscience.

[66]  Margaret L. Schlichting,et al.  Learning-related representational changes reveal dissociable integration and separation signatures in the hippocampus and prefrontal cortex , 2015, Nature Communications.

[67]  Robin I. M. Dunbar,et al.  Processing of different spatial scales in the human brain , 2019, eLife.

[68]  Dagmar Zeithamova,et al.  Flexible Memories: Differential Roles for Medial Temporal Lobe and Prefrontal Cortex in Cross-Episode Binding , 2010, The Journal of Neuroscience.

[69]  G. Schoenbaum,et al.  Model‐based learning and the contribution of the orbitofrontal cortex to the model‐free world , 2012, The European journal of neuroscience.

[70]  M. Botvinick,et al.  The successor representation in human reinforcement learning , 2016, Nature Human Behaviour.

[71]  James K. Kroger,et al.  Rostrolateral Prefrontal Cortex Involvement in Relational Integration during Reasoning , 2001, NeuroImage.

[72]  Jian Xu,et al.  The Anterior Hippocampus Supports a Coarse, Global Environmental Representation and the Posterior Hippocampus Supports Fine-grained, Local Environmental Representations , 2013, Journal of Cognitive Neuroscience.

[73]  David J Foster,et al.  Hippocampal Replay Captures the Unique Topological Structure of a Novel Environment , 2014, The Journal of Neuroscience.

[74]  J. Gabrieli,et al.  The frontopolar cortex and human cognition: Evidence for a rostrocaudal hierarchical organization within the human prefrontal cortex , 2000, Psychobiology.

[75]  M. Moser,et al.  Pattern Separation in the Dentate Gyrus and CA3 of the Hippocampus , 2007, Science.

[76]  Per B. Brockhoff,et al.  lmerTest Package: Tests in Linear Mixed Effects Models , 2017 .

[77]  Nicholas B. Turk-Browne,et al.  Complementary learning systems within the hippocampus: A neural network modeling approach to reconciling episodic memory with statistical learning , 2016, bioRxiv.

[78]  Morris Moscovitch,et al.  Strategic retrieval and the frontal lobes: Evidence from confabulation and amnesia , 1997, Neuropsychologia.

[79]  Peter Gärdenfors,et al.  Navigating cognition: Spatial codes for human thinking , 2018, Science.

[80]  D. Hassabis,et al.  Neural Mechanisms of Hierarchical Planning in a Virtual Subway Network , 2016, Neuron.

[81]  E. Maguire,et al.  The Human Hippocampus and Spatial and Episodic Memory , 2002, Neuron.

[82]  Brice A. Kuhl,et al.  Overlap among Spatial Memories Triggers Repulsion of Hippocampal Representations , 2017, Current Biology.

[83]  Beatrix Emo,et al.  Hippocampal and prefrontal processing of network topology to simulate the future , 2017, Nature Communications.

[84]  M. Roesch,et al.  Effects of Inference on Dopaminergic Prediction Errors Depend on Orbitofrontal Processing , 2017, Behavioral neuroscience.

[85]  Etienne Koechlin,et al.  Frontal pole function: what is specifically human? , 2011, Trends in Cognitive Sciences.

[86]  M. Fyhn,et al.  Progressive increase in grid scale from dorsal to ventral medial entorhinal cortex , 2008, Hippocampus.

[87]  Morris Moscovitch,et al.  Details, gist and schema: hippocampal–neocortical interactions underlying recent and remote episodic and spatial memory , 2017, Current Opinion in Behavioral Sciences.

[88]  Timothy E. J. Behrens,et al.  Counterfactual Choice and Learning in a Neural Network Centered on Human Lateral Frontopolar Cortex , 2011, PLoS biology.

[89]  Nachum Ulanovsky,et al.  Vectorial representation of spatial goals in the hippocampus of bats , 2017, Science.

[90]  Wolfgang M. Pauli,et al.  Evidence for model-based encoding of Pavlovian contingencies in the human brain , 2019, Nature Communications.

[91]  Margaret L. Schlichting,et al.  The Hippocampus and Memory Integration: Building Knowledge to Navigate Future Decisions , 2017 .

[92]  Geoffrey Schoenbaum,et al.  Learning theory: A driving force in understanding orbitofrontal function , 2014, Neurobiology of Learning and Memory.

[93]  Soyoung Q. Park,et al.  The neural code of reward anticipation in human orbitofrontal cortex , 2010, Proceedings of the National Academy of Sciences.

[94]  Neural Mechanisms Underlying Schemas and Inferences , 2018, The Journal of Neuroscience.

[95]  L. Nadel,et al.  The Hippocampus as a Cognitive Map , 1978 .

[96]  P. Caroni,et al.  Goal-oriented searching mediated by ventral hippocampus early in trial-and-error learning , 2012, Nature Neuroscience.

[97]  Margaret L. Schlichting,et al.  Memory integration: neural mechanisms and implications for behavior , 2015, Current Opinion in Behavioral Sciences.

[98]  Andrea Bergmann,et al.  Statistical Parametric Mapping The Analysis Of Functional Brain Images , 2016 .

[99]  Martin Llofriu,et al.  The ventral hippocampus is involved in multi‐goal obstacle‐rich spatial navigation , 2018, Hippocampus.

[100]  M. Botvinick,et al.  The hippocampus as a predictive map , 2016 .

[101]  Jason Wittenberg,et al.  Clarify: Software for Interpreting and Presenting Statistical Results , 2003 .

[102]  Russell A. Epstein,et al.  The cognitive map in humans: spatial navigation and beyond , 2017, Nature Neuroscience.

[103]  P. Dayan,et al.  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control , 2005, Nature Neuroscience.

[104]  Jörn Diedrichsen,et al.  Reliability of dissimilarity measures for multi-voxel pattern analysis , 2016, NeuroImage.

[105]  Robert E. Gross,et al.  Single-Neuron Representations of Spatial Targets in Humans , 2019, Current Biology.

[106]  Michael W. Cole,et al.  Rapid Transfer of Abstract Rules to Novel Contexts in Human Lateral Prefrontal Cortex , 2011, Front. Hum. Neurosci..

[107]  S. Phinn,et al.  Australian vegetated coastal ecosystems as global hotspots for climate change mitigation , 2019, Nature Communications.

[108]  Valerie A. Carr,et al.  Prospective representation of navigational goals in the human hippocampus , 2016, Science.

[109]  John-Dylan Haynes,et al.  Human anterior prefrontal cortex encodes the ‘what’ and ‘when’ of future intentions , 2012, NeuroImage.