Interactions of spatial strategies producing generalization gradient and blocking: A computational approach

We present a computational model of spatial navigation comprising different learning mechanisms in mammals, i.e., associative, cognitive mapping and parallel systems. This model is able to reproduce a large number of experimental results in different variants of the Morris water maze task, including standard associative phenomena (spatial generalization gradient and blocking), as well as navigation based on cognitive mapping. Furthermore, we show that competitive and cooperative patterns between different navigation strategies in the model allow to explain previous apparently contradictory results supporting either associative or cognitive mechanisms for spatial learning. The key computational mechanism to reconcile experimental results showing different influences of distal and proximal cues on the behavior, different learning times, and different abilities of individuals to alternatively perform spatial and response strategies, relies in the dynamic coordination of navigation strategies, whose performance is evaluated online with a common currency through a modular approach. We provide a set of concrete experimental predictions to further test the computational model. Overall, this computational work sheds new light on inter-individual differences in navigation learning, and provides a formal and mechanistic approach to test various theories of spatial cognition in mammals.

[1]  Mehdi Khamassi,et al.  Complementary roles of the rat prefrontal cortex and striatum in reward-based learning and shifting navigation strategies. (Rôles complémentaires du cortex préfrontal et du striatum dans l'apprentissage et le changement de stratégies de navigation basées sur la récompense chez le rat) , 2007 .

[2]  Jean-Baptiste Mouret,et al.  Micro-Data Learning: The Other End of the Spectrum , 2016, ERCIM News.

[3]  Angelo Arleo,et al.  Spatial Learning and Action Planning in a Prefrontal Cortical Network Model , 2011, PLoS Comput. Biol..

[4]  Neil Burgess,et al.  Distinct error-correcting and incidental learning of location relative to landmarks and boundaries , 2008, Proceedings of the National Academy of Sciences.

[5]  Mehdi Khamassi,et al.  Actor–Critic Models of Reinforcement Learning in the Basal Ganglia: From Natural to Artificial Rats , 2005, Adapt. Behav..

[6]  Jeansok J Kim,et al.  Multiple brain-memory systems: the whole does not equal the sum of its parts , 2001, Trends in Neurosciences.

[7]  Peter Redgrave,et al.  Layered Control Architectures in Robots and Vertebrates , 1999, Adapt. Behav..

[8]  T. Prescott,et al.  The ventral basal ganglia, a selection mechanism at the crossroads of space, strategy, and reward. , 2010, Progress in Neurobiology.

[9]  Mehdi Khamassi,et al.  Design of a Control Architecture for Habit Learning in Robots , 2014, Living Machines.

[10]  Debbie M. Kelly,et al.  Evidence against integration of spatial maps in humans: generality across real and virtual environments , 2009, Animal Cognition.

[11]  R. J. McDonald,et al.  Multiple Parallel Memory Systems in the Brain of the Rat , 2002, Neurobiology of Learning and Memory.

[12]  J. O’Keefe,et al.  Neuronal computations underlying the firing of place cells and their role in navigation , 1996, Hippocampus.

[13]  Amir Dezfouli,et al.  Speed/Accuracy Trade-Off between the Habitual and the Goal-Directed Processes , 2011, PLoS Comput. Biol..

[14]  N. Daw.  Dopamine: at the intersection of reward and action , 2007, Nature Neuroscience.

[15]  Nicolas W. Schuck,et al.  Human Orbitofrontal Cortex Represents a Cognitive Map of State Space , 2016, Neuron.

[16]  John L Kubie,et al.  Heading‐vector navigation based on head‐direction cells and path integration , 2009, Hippocampus.

[17]  A. Bennett,et al.  Do animals have cognitive maps? , 1996, The Journal of experimental biology.

[18]  R. J. McDonald,et al.  Parallel information processing in the water maze: evidence for independent memory systems involving dorsal striatum and hippocampus. , 1994, Behavioral and neural biology.

[19]  Mehdi Khamassi,et al.  Which Temporal Difference Learning Algorithm Best Reproduces Dopamine Activity in a Multi-choice Task? , 2012, SAB.

[20]  Ricardo Chavarriaga,et al.  Path planning versus cue responding: a bio-inspired model of switching between navigation strategies , 2010, Biological Cybernetics.

[21]  J. Jankowski,et al.  Distinct striatal regions for planning and executing novel and automated movement sequences , 2009, NeuroImage.

[22]  B Poucet,et al.  Medial prefrontal lesions in the rat and spatial navigation: evidence for impaired planning. , 1995, Behavioral neuroscience.

[23]  J. Pearce,et al.  Blocking in the Morris swimming pool. , 1999, Journal of experimental psychology. Animal behavior processes.

[24]  M. Khamassi,et al.  Hippocampal replays under the scrutiny of reinforcement learning models. , 2018, Journal of neurophysiology.

[25]  Ricardo Chavarriaga,et al.  Robust self-localisation and navigation based on hippocampal place cells , 2005, Neural Networks.

[26]  R. Clark,et al.  The Hippocampus and Spatial Memory: Findings with a Novel Modification of the Water Maze , 2007, The Journal of Neuroscience.

[27]  David J. Foster,et al.  A model of hippocampally dependent navigation, using the temporal difference learning rule , 2000, Hippocampus.

[28]  Sidney I. Wiener,et al.  Lesions of the medial shell of the nucleus accumbens impair rats in finding larger rewards, but spare reward-seeking behavior , 2000, Behavioural Brain Research.

[29]  R. O’Reilly,et al.  Separate neural substrates for skill learning and performance in the ventral and dorsal striatum , 2007, Nature Neuroscience.

[30]  N. Burgess,et al.  Complementary memory systems: competition, cooperation and compensation , 2005, Trends in Neurosciences.

[31]  Ricardo Chavarriaga,et al.  A Computational Model of Parallel Navigation Systems in Rodents , 2005 .

[32]  N. White.  The role of stimulus ambiguity and movement in spatial navigation: A multiple memory systems analysis of location discrimination , 2004, Neurobiology of Learning and Memory.

[33]  Christian F. Doeller,et al.  Parallel striatal and hippocampal systems for landmarks and boundaries in spatial memory , 2008, Proceedings of the National Academy of Sciences.

[34]  Matthijs A. A. van der Meer,et al.  Ventral striatum: a critical look at models of learning and evaluation , 2011, Current Opinion in Neurobiology.

[35]  Angelo Arleo,et al.  Spatial cognition and neuro-mimetic navigation: a model of hippocampal place cell activity , 2000, Biological Cybernetics.

[36]  C. Gerfen,et al.  CHAPTER 18 – Basal Ganglia , 2004 .

[37]  A. D. Redish,et al.  Task-dependent encoding of space and events by striatal neurons is dependent on neural subtype , 2008, Neuroscience.

[38]  R. J. McDonald,et al.  Multiple memory systems: The power of interactions , 2004, Neurobiology of Learning and Memory.

[39]  Timothy E. J. Behrens,et al.  Organizing conceptual knowledge in humans with a gridlike code , 2016, Science.

[40]  M. Wilson,et al.  VTA neurons coordinate with the hippocampal reactivation of spatial experience , 2015, eLife.

[41]  J. O'Keefe,et al.  The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat. , 1971, Brain research.

[42]  Victoria D. Chamizo,et al.  Generalization gradients in a navigation task with rats , 2006 .

[43]  Ricardo Chavarriaga,et al.  Analyzing Interactions between Cue-Guided and Place-Based Navigation with a Computational Model of Action Selection: Influence of Sensory Cues and Training , 2010, SAB.

[44]  A. Redish.  Beyond the Cognitive Map: From Place Cells to Episodic Memory , 1999 .

[45]  N. Daw,et al.  Integrating memories to guide decisions , 2015, Current Opinion in Behavioral Sciences.

[46]  John M. Pearce,et al.  Hippocampal lesions disrupt navigation based on cognitive maps but not heading vectors , 1998, Nature.

[47]  Ricardo Chavarriaga,et al.  Analyzing Interactions between Navigation Strategies Using a Computational Model of Action Selection , 2008, Spatial Cognition.

[48]  J. Watson.  Psychology As The Behaviorist Views It , 2011 .

[49]  Aaron P Blaisdell,et al.  Associative Basis of Landmark Learning and Integration in Vertebrates. , 2009, Comparative cognition & behavior reviews.

[50]  Paul E. Gold,et al.  Coordination of multiple memory systems , 2004, Neurobiology of Learning and Memory.

[51]  V. Brown,et al.  Rodent models of prefrontal cortical function , 2002, Trends in Neurosciences.

[52]  G. Buzsáki,et al.  Selective suppression of hippocampal ripples impairs spatial memory , 2009, Nature Neuroscience.

[53]  J. Pearce,et al.  The 36th Sir Frederick Bartlett Lecture: An associative analysis of spatial learning , 2009, Quarterly journal of experimental psychology.

[54]  B. Knowlton,et al.  Contributions of striatal subregions to place and response learning. , 2004, Learning & memory.

[55]  D. M. Skinner,et al.  An analysis of response, direction, and place learning in an open field and T maze. , 2003, Journal of experimental psychology. Animal behavior processes.

[56]  P. Dayan,et al.  Choice values , 2006, Nature Neuroscience.

[57]  Tamás Kiss,et al.  Episodes in Space: A Modeling Study of Hippocampal Place Representation , 2008, SAB.

[58]  J. D. McGaugh,et al.  Inactivation of Hippocampus or Caudate Nucleus with Lidocaine Differentially Affects Expression of Place and Response Learning , 1996, Neurobiology of Learning and Memory.

[59]  R. Rescorla.  A theory of pavlovian conditioning: The effectiveness of reinforcement and non-reinforcement , 1972 .

[60]  N. White,et al.  Parallel Information Processing in the Dorsal Striatum: Relation to Hippocampal Function , 1999, The Journal of Neuroscience.

[61]  H. Yin,et al.  The role of the basal ganglia in habit formation , 2006, Nature Reviews Neuroscience.

[62]  R. J. McDonald,et al.  The challenges of understanding mammalian cognition and memory-based behaviours: an interactive learning and memory systems approach , 2004, Neuroscience & Biobehavioral Reviews.

[63]  James J Knierim,et al.  Distal landmarks and hippocampal place cells: Effects of relative translation versus rotation , 2003, Hippocampus.

[64]  Eytan Ruppin,et al.  Actor-critic models of the basal ganglia: new anatomical and computational perspectives , 2002, Neural Networks.

[65]  P. E. Gold,et al.  Switching Memory Systems during Learning: Changes in Patterns of Brain Acetylcholine Release in the Hippocampus and Striatum in Rats , 2003, The Journal of Neuroscience.

[66]  Mehdi Khamassi,et al.  Modeling choice and reaction time during arbitrary visuomotor learning through the coordination of adaptive working memory and reinforcement learning , 2015, Front. Behav. Neurosci..

[67]  Mehdi Khamassi,et al.  Modelling Individual Differences in the Form of Pavlovian Conditioned Approach Responses: A Dual Learning Systems Approach with Factored Representations , 2014, PLoS Comput. Biol..

[68]  M. Roesch,et al.  Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards , 2007, Nature Neuroscience.

[69]  V. Srinivasa Chakravarthy,et al.  What do the basal ganglia do? A modeling perspective , 2010, Biological Cybernetics.

[70]  W. Gerstner,et al.  Stress, genotype and norepinephrine in the prediction of mouse behavior using reinforcement learning , 2009, Nature Neuroscience.

[71]  Timothy E. J. Behrens,et al.  Learning the value of information in an uncertain world , 2007, Nature Neuroscience.

[72]  Jean-Arcady Meyer,et al.  BIOLOGICALLY BASED ARTIFICIAL NAVIGATION SYSTEMS: REVIEW AND PROSPECTS , 1997, Progress in Neurobiology.

[73]  H. Eichenbaum.  Prefrontal–hippocampal interactions in episodic memory , 2017, Nature Reviews Neuroscience.

[74]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[75]  E. Tolman.  Cognitive maps in rats and men. , 1948, Psychological review.

[76]  J. D. McGaugh,et al.  Double dissociation of fornix and caudate nucleus lesions on acquisition of two water maze tasks: further evidence for multiple memory systems. , 1992, Behavioral neuroscience.

[77]  R U Muller,et al.  Head-direction cells recorded from the postsubiculum in freely moving rats. I. Description and quantitative analysis , 1990, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[78]  M. Shapiro,et al.  A Map for Social Navigation in the Human Brain , 2015, Neuron.

[79]  R. Poldrack,et al.  Competition among multiple memory systems: converging evidence from animal and human brain studies , 2003, Neuropsychologia.

[80]  R. D'Hooge,et al.  Applications of the Morris water maze in the study of learning and memory , 2001, Brain Research Reviews.

[81]  W. Schultz,et al.  The phasic dopamine signal maturing: from reward via behavioural activation to formal economic utility , 2017, Current Opinion in Neurobiology.

[82]  Elena Papaleo,et al.  An Acidic Loop and Cognate Phosphorylation Sites Define a Molecular Switch That Modulates Ubiquitin Charging Activity in Cdc34-Like Enzymes , 2011, PLoS Comput. Biol..

[83]  P. Redgrave,et al.  The basal ganglia: a vertebrate solution to the selection problem? , 1999, Neuroscience.

[84]  E. Vaadia,et al.  Midbrain dopamine neurons encode decisions for future action , 2006, Nature Neuroscience.

[85]  Michael A. Arbib,et al.  Affordances. Motivations, and the World Graph Theory , 1998, Adapt. Behav..

[86]  Matthijs A. A. van der Meer,et al.  Information Processing in Decision-Making Systems , 2012, The Neuroscientist : a review journal bringing neurobiology, neurology and psychiatry.

[87]  D. Shohamy,et al.  Feedback Timing Modulates Brain Systems for Learning in Humans , 2011, The Journal of Neuroscience.

[88]  Anne G E Collins,et al.  How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis , 2012, The European journal of neuroscience.

[89]  Debbie M. Kelly,et al.  Spatial navigation: Spatial learning in real and virtual environments , 2006 .

[90]  M. Khamassi,et al.  Integrating cortico-limbic-basal ganglia architectures for learning model-based and model-free navigation strategies , 2012, Front. Behav. Neurosci..

[91]  K Caluwaerts,et al.  A biologically inspired meta-control navigation system for the Psikharpax rat robot , 2012, Bioinspiration & biomimetics.

[92]  Travis E. Johnson,et al.  The relative influence of place and direction in the Morris water task. , 2008, Journal of experimental psychology. Animal behavior processes.

[93]  Edsger W. Dijkstra,et al.  A note on two problems in connexion with graphs , 1959, Numerische Mathematik.

[94]  N. White.  Some highlights of research on the effects of caudate nucleus lesions over the past 200 years , 2009, Behavioural Brain Research.

[95]  Samuel M. McClure,et al.  Hierarchical control over effortful behavior by rodent medial frontal cortex: A computational model. , 2015, Psychological review.

[96]  D. Shohamy,et al.  Integrating Memories in the Human Brain: Hippocampal-Midbrain Encoding of Overlapping Events , 2008, Neuron.

[97]  P. Dayan,et al.  States versus Rewards: Dissociable Neural Prediction Error Signals Underlying Model-Based and Model-Free Reinforcement Learning , 2010, Neuron.

[98]  Mehdi Khamassi,et al.  Adaptive coordination of working-memory and reinforcement learning in non-human primates performing a trial-and-error problem solving task , 2017, Behavioural Brain Research.

[99]  Mehdi Khamassi,et al.  Coherent Theta Oscillations and Reorganization of Spike Timing in the Hippocampal- Prefrontal Network upon Learning , 2010, Neuron.

[100]  Jessica A. Grahn,et al.  The cognitive functions of the caudate nucleus , 2008, Progress in Neurobiology.

[101]  P. Dayan,et al.  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control , 2005, Nature Neuroscience.

[102]  M. Frank,et al.  Anatomy of a decision: striato-orbitofrontal interactions in reinforcement learning, decision making, and reversal. , 2006, Psychological review.

[103]  Mehdi Khamassi,et al.  Combining Self-organizing Maps with Mixtures of Experts: Application to an Actor-Critic Model of Reinforcement Learning in the Basal Ganglia , 2006, SAB.

[104]  D. Shohamy,et al.  The role of the basal ganglia in learning and memory: Insight from Parkinson’s disease , 2011, Neurobiology of Learning and Memory.

[105]  B. Poucet.  Spatial cognitive maps in animals: new hypotheses on their structure and neural mechanisms. , 1993, Psychological review.

[106]  John B. Watson,et al.  Psychology as the behaviorist views it, 1913. , 1948 .

[107]  S. Grillner,et al.  Evolutionary Conservation of the Basal Ganglia as a Common Vertebrate Mechanism for Action Selection , 2011, Current Biology.

[108]  B. Balleine,et al.  The role of the dorsomedial striatum in instrumental conditioning , 2005, The European journal of neuroscience.

[109]  N. Daw,et al.  Multiplicity of control in the basal ganglia: computational roles of striatal subregions , 2011, Current Opinion in Neurobiology.

[110]  Anthony McGregor,et al.  Absence of an interaction between navigational strategies based on local and distal landmarks. , 2004, Journal of experimental psychology. Animal behavior processes.

[111]  Travis E. Johnson,et al.  Evidence for a shift from place navigation to directional responding in one variant of the Morris water task. , 2009, Journal of experimental psychology. Animal behavior processes.

[112]  B. Gibson,et al.  Cognitive maps not used by humans (Homo sapiens) during a dynamic navigational task. , 2001, Journal of comparative psychology.

[113]  R. Morris.  Spatial Localization Does Not Require the Presence of Local Cues , 1981 .

[114]  Guillén Fernández,et al.  Interaction between the Human Hippocampus and the Caudate Nucleus during Route Recognition , 2004, Neuron.

[115]  R. Morris,et al.  Delay‐dependent impairment of a matching‐to‐place task with chronic and intrahippocampal infusion of the NMDA‐antagonist D‐AP5 , 1999, Hippocampus.

[116]  P. E. Gold,et al.  Acetylcholine release in the hippocampus and striatum during place and response training. , 2005, Learning & memory.

[117]  R. Morris,et al.  Place navigation impaired in rats with hippocampal lesions , 1982, Nature.

[118]  Jean-Arcady Meyer,et al.  Integration of Navigation and Action Selection Functionalities in a Computational Model of Cortico-Basal-Ganglia–Thalamo-Cortical Loops , 2005, Adapt. Behav..

[119]  Mitsuo Kawato,et al.  Multiple Model-Based Reinforcement Learning , 2002, Neural Computation.

[120]  Giovanni Pezzulo,et al.  The Mixed Instrumental Controller: Using Value of Information to Combine Habitual Choice and Mental Simulation , 2013, Front. Psychol..

[121]  A. Graybiel,et al.  Differential Dynamics of Activity Changes in Dorsolateral and Dorsomedial Striatal Loops during Learning , 2010, Neuron.

[122]  M. Packard,et al.  Factors that influence the relative use of multiple memory systems , 2013, Hippocampus.

[123]  R. Kesner,et al.  Involvement of the Prelimbic–Infralimbic Areas of the Rodent Prefrontal Cortex in Behavioral Flexibility for Place and Response Learning , 1999, The Journal of Neuroscience.

[124]  R. M. Elliott,et al.  Behavior of Organisms , 1991 .