Dorsal hippocampus contributes to model-based planning

Planning can be defined as action selection that leverages an internal model of the outcomes likely to follow each possible action. Its neural mechanisms remain poorly understood. Here we adapt recent advances from human research for rats, presenting for the first time an animal task that produces many trials of planned behavior per session, making multitrial rodent experimental tools available to study planning. We use part of this toolkit to address a perennially controversial issue in planning: the role of the dorsal hippocampus. Although prospective hippocampal representations have been proposed to support planning, intact planning in animals with damaged hippocampi has been repeatedly observed. Combining formal algorithmic behavioral analysis with muscimol inactivation, we provide causal evidence directly linking dorsal hippocampus with planning behavior. Our results and methods open the door to new and more detailed investigations of the neural mechanisms of planning in the hippocampus and throughout the brain.

[1]  Jiqiang Guo,et al.  Stan: A Probabilistic Programming Language. , 2017, Journal of statistical software.

[2]  Matthijs Verhage,et al.  A solution to dependency: using multilevel analysis to accommodate nested data , 2014, Nature Neuroscience.

[3]  J. O’Keefe,et al.  Space in the brain: how the hippocampal formation supports spatial cognition , 2014, Philosophical Transactions of the Royal Society B: Biological Sciences.

[4]  P. Dayan,et al.  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control , 2005, Nature Neuroscience.

[5]  Nic Wilson,et al.  A Probabilistic Programming Language for Influence Diagrams , 2017, SUM.

[6]  David B. Dunson,et al.  Bayesian Data Analysis , 2010 .

[7]  Robert C. Wilson,et al.  Orbitofrontal Cortex as a Cognitive Map of Task Space , 2014, Neuron.

[8]  Robert Colin Honey,et al.  Excitotoxic lesions of the hippocampus leave sensory preconditioning intact: Implications for models of hippocampal function , 2001 .

[9]  D. Hassabis,et al.  Deconstructing episodic memory with construction , 2007, Trends in Cognitive Sciences.

[10]  Brad E. Pfeiffer,et al.  Hippocampal place cell sequences depict future paths to remembered goals , 2013, Nature.

[11]  Y. Niv,et al.  Ventral Striatum and Orbitofrontal Cortex Are Both Required for Model-Based, But Not Model-Free, Reinforcement Learning , 2011, The Journal of Neuroscience.

[12]  G. Handelmann,et al.  Hippocampus, space, and memory , 1979 .

[13]  Peter Dayan,et al.  Simple Plans or Sophisticated Habits? State, Transition and Learning Interactions in the Two-step Task , 2015, bioRxiv.

[14]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[15]  E. Tolman.  Cognitive maps in rats and men. , 1948, Psychological review.

[16]  B. Balleine,et al.  Sensitivity to Instrumental Contingency Degradation Is Mediated by the Entorhinal Cortex and Its Efferents via the Dorsal Hippocampus , 2002, The Journal of Neuroscience.

[17]  D. Kimble,et al.  Latent learning in hippocampal-lesioned rats , 1981, Physiology & Behavior.

[18]  G. Schoenbaum,et al.  What the orbitofrontal cortex does not do , 2015, Nature Neuroscience.

[19]  R. Racine,et al.  Hippocampal lesions and delayed alternation in the rat , 1965 .

[20]  H. Eichenbaum,et al.  Memory for the Order of Events in Specific Sequences: Contributions of the Hippocampus and Medial Prefrontal Cortex , 2011, The Journal of Neuroscience.

[21]  R. Costa,et al.  Endocannabinoid Signaling is Critical for Habit Formation , 2007, Frontiers in integrative neuroscience.

[22]  Matthijs A. A. van der Meer,et al.  Internally generated sequences in learning and executing goal-directed behavior , 2014, Trends in Cognitive Sciences.

[23]  Joshua L. Jones,et al.  Orbitofrontal Cortex Supports Behavior and Learning Using Inferred But Not Cached Values , 2012, Science.

[24]  G. Winocur,et al.  Higher-Order Conditioning Is Impaired by Hippocampal Lesions , 2014, Current Biology.

[25]  Carlos D. Brody,et al.  Identifying Model-Based and Model-Free Patterns in Behavior on Multi-Step Tasks , 2016, bioRxiv.

[26]  R. F. Thompson,et al.  Hippocampus and trace conditioning of the rabbit's classically conditioned nictitating membrane response. , 1986, Behavioral neuroscience.

[27]  David J. Foster,et al.  Sequence learning and the role of the hippocampus in rodent navigation , 2012, Current Opinion in Neurobiology.

[28]  Wouter Kool,et al.  When Does Model-Based Control Pay Off? , 2016, PLoS Comput. Biol..

[29]  H. Eichenbaum,et al.  Can We Reconcile the Declarative Memory and Spatial Navigation Views on Hippocampal Function? , 2014, Neuron.

[30]  David J. Foster,et al.  A model of hippocampally dependent navigation, using the temporal difference learning rule , 2000, Hippocampus.

[31]  Peter Dayan,et al.  Interplay of approximate planning strategies , 2015, Proceedings of the National Academy of Sciences.

[32]  W. Brogden.  Sensory pre-conditioning. , 1939 .

[33]  B. Balleine,et al.  Human and Rodent Homologies in Action Control: Corticostriatal Determinants of Goal-Directed and Habitual Action , 2010, Neuropsychopharmacology.

[34]  Christopher D. Adams,et al.  Instrumental Responding following Reinforcer Devaluation , 1981 .

[35]  R. Buckner.  The role of the hippocampus in prediction and imagination. , 2010, Annual review of psychology.

[36]  R. C. Honey,et al.  Excitotoxic lesions of the hippocampus leave sensory preconditioning intact: implications for models of hippocampal function. , 2001, Behavioral neuroscience.

[37]  P. Dudchenko.  The hippocampus as a cognitive map , 2010 .

[38]  H. Eichenbaum,et al.  The hippocampus and memory for orderly stimulus relations. , 1997, Proceedings of the National Academy of Sciences of the United States of America.

[39]  S. Duane,et al.  Hybrid Monte Carlo , 1987 .

[40]  D. Kimble,et al.  Further evidence for latent learning in hippocampal-lesioned rats , 1982, Physiology & Behavior.

[41]  C. Padoa-Schioppa.  Neurobiology of economic choice: a good-based model. , 2011, Annual review of neuroscience.

[42]  P. Dayan,et al.  Goals and Habits in the Brain , 2013, Neuron.

[43]  P. Dayan,et al.  Mapping value based planning and extensively trained choice in the human brain , 2012, Nature Neuroscience.

[44]  H. Eichenbaum,et al.  Conservation of hippocampal memory function in rats and humans , 1996, Nature.

[45]  N. White,et al.  Inactivation of the dorsal hippocampus does not affect learning during exploration of a novel environment , 2005, Hippocampus.

[46]  P. Dayan,et al.  Model-based influences on humans’ choices and striatal prediction errors , 2011, Neuron.

[47]  B. Balleine,et al.  Orbitofrontal Cortex Mediates Outcome Encoding in Pavlovian But Not Instrumental Conditioning , 2007, The Journal of Neuroscience.

[48]  John K Kruschke,et al.  Bayesian data analysis. , 2010, Wiley interdisciplinary reviews. Cognitive science.

[49]  L. Nadel,et al.  The Hippocampus as a Cognitive Map , 1978 .

[50]  R. Morris,et al.  Place navigation impaired in rats with hippocampal lesions , 1982, Nature.

[51]  M. Nicolelis,et al.  Immediate thalamic sensory plasticity depends on corticothalamic feedback. , 1999, Proceedings of the National Academy of Sciences of the United States of America.

[52]  Zeb Kurth-Nelson,et al.  Model-Based Reasoning in Humans Becomes Automatic with Training , 2015, PLoS Comput. Biol..

[53]  Guido Bugmann,et al.  Effects of medial septal lesions: implications for models of hippocampal function , 1997 .

[54]  B. Balleine,et al.  The Role of the Hippocampus in Instrumental Conditioning , 2000, The Journal of Neuroscience.

[55]  J. D. McGaugh,et al.  Inactivation of Hippocampus or Caudate Nucleus with Lidocaine Differentially Affects Expression of Place and Response Learning , 1996, Neurobiology of Learning and Memory.

[56]  Andrew M. Wikenheiser,et al.  Hippocampal theta sequences reflect current goals , 2015, Nature Neuroscience.

[57]  T. Robbins,et al.  Decision Making, Affect, and Learning: Attention and Performance XXIII , 2011 .

[58]  L. Jarrard,et al.  Contributions of the hippocampus and medial prefrontal cortex to energy and body weight regulation , 2009, Hippocampus.

[59]  J. O'Keefe,et al.  The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat. , 1971, Brain research.

[60]  Michael E. Hasselmo,et al.  Modeling goal-directed spatial navigation in the rat based on physiological data from the hippocampal formation , 2003, Neural Networks.

[61]  P. Glimcher,et al.  JOURNAL OF THE EXPERIMENTAL ANALYSIS OF BEHAVIOR 2005, 84, 555–579 NUMBER 3(NOVEMBER) DYNAMIC RESPONSE-BY-RESPONSE MODELS OF MATCHING BEHAVIOR IN RHESUS MONKEYS , 2022 .

[62]  John H. Martin.  Autoradiographic estimation of the extent of reversible inactivation produced by microinjection of lidocaine and muscimol in the rat , 1991, Neuroscience Letters.

[63]  Dylan A. Simon,et al.  Neural Correlates of Forward Planning in a Spatial Decision Task in Humans , 2011, The Journal of Neuroscience.

[64]  Amir Dezfouli,et al.  Speed/Accuracy Trade-Off between the Habitual and the Goal-Directed Processes , 2011, PLoS Comput. Biol..

[65]  R. Costa,et al.  Orbitofrontal and striatal circuits dynamically encode the shift between goal-directed and habitual actions , 2013, Nature Communications.