A reinforcement learning approach to model interactions between landmarks and geometric cues during spatial learning

In contrast to predictions derived from the associative learning theory, a number of behavioral studies suggested the absence of competition between geometric cues and landmarks in some experimental paradigms. In parallel to these studies, neurobiological experiments suggested the existence of separate independent memory systems which may not always interact according to classic associative principles. In this paper we attempt to combine these two lines of research by proposing a model of spatial learning that is based on the theory of multiple memory systems. In our model, a place-based locale strategy uses activities of modeled hippocampal place cells to drive navigation to a hidden goal, while a stimulus-response taxon strategy, presumably mediated by the dorso-lateral striatum, learns landmark-approaching behavior. A strategy selection network, proposed to reside in the prefrontal cortex, implements a simple reinforcement learning rule to switch behavioral strategies. The model is used to reproduce the results of a behavioral experiment in which an interaction between a landmark and geometric cues was studied. We show that this model, built on the basis of neurobiological data, can explain the lack of competition between the landmark and geometry, potentiation of geometry learning by the landmark, and blocking. Namely, we propose that the geometry potentiation is a consequence of cooperation between memory systems during learning, while blocking is due to competition between the memory systems during action selection.

[1]  A. Redish Beyond the Cognitive Map: From Place Cells to Episodic Memory , 1999 .

[2]  Sara J Shettleworth,et al.  The geometric module in the rat: independence of shape and feature learning in a food finding task , 2004, Learning & behavior.

[3]  K. Cheng A purely geometric module in the rat's spatial representation , 1986, Cognition.

[4]  J. Pearce,et al.  Failure of a landmark to restrict spatial learning based on the shape of the environment , 2004, The Quarterly journal of experimental psychology. B, Comparative and physiological psychology.

[5]  R. Kesner,et al.  Involvement of the Prelimbic–Infralimbic Areas of the Rodent Prefrontal Cortex in Behavioral Flexibility for Place and Response Learning , 1999, The Journal of Neuroscience.

[6]  Ricardo Chavarriaga,et al.  Path planning versus cue responding: a bio-inspired model of switching between navigation strategies , 2010, Biological Cybernetics.

[7]  J. O’Keefe,et al.  Geometric determinants of the place fields of hippocampal neurons , 1996, Nature.

[8]  Ziv M. Williams,et al.  Selective enhancement of associative learning by microstimulation of the anterior caudate , 2006, Nature Neuroscience.

[9]  Erin L. Rich,et al.  Rat Prefrontal Cortical Neurons Selectively Code Strategy Switches , 2009, The Journal of Neuroscience.

[10]  K. Jeffery,et al.  Please Scroll down for Article the Quarterly Journal of Experimental Psychology Theoretical Accounts of Spatial Learning: a Neurobiological View(commentary on Pearce, 2009) , 2022 .

[11]  S. Shettleworth,et al.  Learning about Environmental Geometry: an Associative Model , 2022 .

[12]  J. D. McGaugh,et al.  Inactivation of Hippocampus or Caudate Nucleus with Lidocaine Differentially Affects Expression of Place and Response Learning , 1996, Neurobiology of Learning and Memory.

[13]  R. J. McDonald,et al.  Multiple Parallel Memory Systems in the Brain of the Rat , 2002, Neurobiology of Learning and Memory.

[14]  N. Newcombe,et al.  Is there a geometric module for spatial orientation? squaring theory and evidence , 2005, Psychonomic bulletin & review.

[15]  E. Tolman Cognitive maps in rats and men. , 1948, Psychological review.

[16]  Peter M. Jones,et al.  Potentiation, overshadowing, and blocking of spatial learning based on the shape of the environment. , 2006, Journal of experimental psychology. Animal behavior processes.

[17]  Guillem R. Esber,et al.  Absence of overshadowing between a landmark and geometric cues in a distinctively shaped environment: a test of Miller and Shettleworth (2007). , 2009, Journal of experimental psychology. Animal behavior processes.

[18]  J. Pearce,et al.  Absence of Overshadowing and Blocking between Landmarks and the Geometric Cues Provided by the Shape of a Test Arena , 2003, The Quarterly journal of experimental psychology. B, Comparative and physiological psychology.

[19]  E. Fischer Conditioned Reflexes , 1942, American journal of physical medicine.

[20]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[21]  Michael R. W. Dawson,et al.  Using perceptrons to explore the reorientation task , 2010, Cognition.

[22]  Ricardo Chavarriaga,et al.  Competition between cue response and place response: a model of rat navigation behaviour , 2005, Connect. Sci..

[23]  Angelo Arleo,et al.  Spatial orientation in navigating agents: Modeling head-direction cells , 2001, Neurocomputing.

[24]  Erin L. Rich,et al.  Prelimbic/Infralimbic Inactivation Impairs Memory for Multiple Task Switches, But Not Flexible Selection of Familiar Tasks , 2007, The Journal of Neuroscience.

[25]  R. Andersen,et al.  Multimodal representation of space in the posterior parietal cortex and its use in planning movements. , 1997, Annual review of neuroscience.

[26]  N. Emery Cognition, Evolution, and Behavior Cognition, Evolution, and Behavior. 2nd edn. By Sara J. Shettleworth. Oxford: Oxford University Press (2009). Pp. xiii+700. Price $59.95 paperback. , 2010, Animal Behaviour.

[27]  Ricardo Chavarriaga,et al.  Minimal Model of Strategy Switching in the Plus-Maze Navigation Task , 2010, SAB.

[28]  L. Kamin Predictability, surprise, attention, and conditioning , 1967 .

[29]  W. Gerstner,et al.  Is there a geometric module for spatial orientation? Insights from a rodent navigation model. , 2009, Psychological review.

[30]  J Ward-Robinson,et al.  Influence of a beacon on spatial learning based on the shape of the test environment. , 2001, Journal of experimental psychology. Animal behavior processes.

[31]  J. Pearce,et al.  The 36th Sir Frederick Bartlett Lecture: An associative analysis of spatial learning , 2009, Quarterly journal of experimental psychology.

[32]  Jeffrey S. Taube,et al.  Head direction cells and the neural mechanisms of spatial orientation , 2005 .

[33]  W. Schultz,et al.  Dopamine responses comply with basic assumptions of formal learning theory , 2001, Nature.

[34]  Christian F. Doeller,et al.  Parallel striatal and hippocampal systems for landmarks and boundaries in spatial memory , 2008, Proceedings of the National Academy of Sciences.

[35]  B. Balleine,et al.  The Role of the Dorsal Striatum in Reward and Decision-Making , 2007, The Journal of Neuroscience.

[36]  Angelo Arleo,et al.  Spatial cognition and neuro-mimetic navigation: a model of hippocampal place cell activity , 2000, Biological Cybernetics.

[37]  Neil Burgess,et al.  Distinct error-correcting and incidental learning of location relative to landmarks and boundaries , 2008, Proceedings of the National Academy of Sciences.

[38]  W. F. Prokasy,et al.  Classical conditioning II: Current research and theory. , 1972 .

[39]  Ricardo Chavarriaga,et al.  A Computational Model of Parallel Navigation Systems in Rodents , 2005 .

[40]  J. Pearce,et al.  Blocking in the Morris swimming pool. , 1999, Journal of experimental psychology. Animal behavior processes.

[41]  R. Rescorla,et al.  A theory of Pavlovian conditioning : Variations in the effectiveness of reinforcement and nonreinforcement , 1972 .

[42]  J. Pearce,et al.  Spatial learning based on the shape of the environment is influenced by properties of the objects forming the shape. , 2006, Journal of experimental psychology. Animal behavior processes.

[43]  B. Campbell,et al.  Punishment and aversive behavior , 1969 .

[44]  Sara J Shettleworth,et al.  An associative model of geometry learning: a modified choice rule. , 2008, Journal of experimental psychology. Animal behavior processes.