Hippocampal Contributions to Control: The Third Way

Recent experimental studies have focused on the specialization of different neural structures for different types of instrumental behavior. Recent theoretical work has provided normative accounts for why there should be more than one control system, and how the output of different controllers can be integrated. Two particlar controllers have been identified, one associated with a forward model and the prefrontal cortex and a second associated with computationally simpler, habitual, actor-critic methods and part of the striatum. We argue here for the normative appropriateness of an additional, but so far marginalized control system, associated with episodic memory, and involving the hippocampus and medial temporal cortices. We analyze in depth a class of simple environments to show that episodic control should be useful in a range of cases characterized by complexity and inferential noise, and most particularly at the very early stages of learning, long before habitization has set in. We interpret data on the transfer of control from the hippocampus to the striatum in the light of this hypothesis.

[1]  G. P. Steck,et al.  Moments of Order Statistics from the Equicorrelated Multivariate Normal Distribution , 1962 .

[2]  Douglas L. Hintzman,et al.  MINERVA 2: A simulation model of human memory , 1984 .

[3]  David L. Waltz,et al.  Toward memory-based reasoning , 1986, CACM.

[4]  J. D. McGaugh,et al.  Double dissociation of fornix and caudate nucleus lesions on acquisition of two water maze tasks: further evidence for multiple memory systems. , 1992, Behavioral neuroscience.

[5]  James L. McClelland,et al.  Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. , 1995, Psychological review.

[6]  Ben J. A. Kröse,et al.  Learning from delayed rewards , 1995, Robotics Auton. Syst..

[7]  Andrew G. Barto,et al.  Reinforcement learning , 1998 .

[8]  A. Dickinson,et al.  Episodic-like memory during cache recovery by scrub jays , 1998, Nature.

[9]  Michael Kearns,et al.  Finite-Sample Convergence Rates for Q-Learning and Indirect Algorithms , 1998, NIPS.

[10]  Rajeev Sharma,et al.  Advances in Neural Information Processing Systems 11 , 1999 .

[11]  M. Gluck,et al.  Interactive memory systems in the human brain , 2001, Nature.

[12]  R. J. McDonald,et al.  Multiple Parallel Memory Systems in the Brain of the Rat , 2002, Neurobiology of Learning and Memory.

[13]  R. Poldrack,et al.  How do memory systems interact? Evidence from human classification learning , 2004, Neurobiology of Learning and Memory.

[14]  Paul Bourgine,et al.  Exploration of Multi-State Environments: Local Measures and Back-Propagation of Uncertainty , 1999, Machine Learning.

[15]  P. Dayan,et al.  Off-line replay maintains declarative memories in a model of hippocampal-neocortical interactions , 2004, Nature Neuroscience.

[16]  P. Dayan,et al.  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control , 2005, Nature Neuroscience.

[17]  Yadin Dudai,et al.  The Janus face of Mnemosyne , 2005, Nature.

[18]  Peter Dayan,et al.  Uncertainty, phase and oscillatory hippocampal recall , 2006, NIPS.

[19]  Konrad Paul Kording,et al.  The dynamics of memory as a consequence of optimal adaptation to a changing body , 2007, Nature Neuroscience.

[20]  Joanna M. Dally,et al.  Social cognition by food-caching corvids. The western scrub-jay as a natural psychologist , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[21]  R. Rosenfeld.  Nature , 2009, Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery.

[22]  Richard S. Sutton,et al.  Reinforcement Learning , 1992, Handbook of Machine Learning.