A biologically inspired meta-control navigation system for the Psikharpax rat robot

A biologically inspired navigation system for the mobile rat-like robot named Psikharpax is presented, allowing for self-localization and autonomous navigation in an initially unknown environment. The ability of parts of the model (e.g. the strategy selection mechanism) to reproduce rat behavioral data in various maze tasks has been validated before in simulations. But the capacity of the model to work on a real robot platform had not been tested. This paper presents our work on the implementation on the Psikharpax robot of two independent navigation strategies (a place-based planning strategy and a cue-guided taxon strategy) and a strategy selection meta-controller. We show how our robot can memorize which was the optimal strategy in each situation, by means of a reinforcement learning algorithm. Moreover, a context detector enables the controller to quickly adapt to changes in the environment-recognized as new contexts-and to restore previously acquired strategy preferences when a previously experienced context is recognized. This produces adaptivity closer to rat behavioral performance and constitutes a computational proposition of the role of the rat prefrontal cortex in strategy shifting. Moreover, such a brain-inspired meta-controller may provide an advancement for learning architectures in robotics.

[1]  B. Kolb,et al.  Do rats have a prefrontal cortex? , 2003, Behavioural Brain Research.

[2]  R. Kesner,et al.  Involvement of the Prelimbic–Infralimbic Areas of the Rodent Prefrontal Cortex in Behavioral Flexibility for Place and Response Learning , 1999, The Journal of Neuroscience.

[3]  A. Redish.  Beyond the Cognitive Map: From Place Cells to Episodic Memory , 1999 .

[4]  Adam Johnson,et al.  Neural Ensembles in CA3 Transiently Encode Paths Forward of the Animal at a Decision Point , 2007, The Journal of Neuroscience.

[5]  R. Pfeifer,et al.  Self-Organization, Embodiment, and Biologically Inspired Robotics , 2007, Science.

[6]  V. Brown,et al.  Medial Frontal Cortex Mediates Perceptual Attentional Set Shifting in the Rat , 2000, The Journal of Neuroscience.

[7]  John M. Pearce,et al.  Hippocampal lesions disrupt navigation based on cognitive maps but not heading vectors , 1998, Nature.

[8]  Angelo Arleo,et al.  Multimodal sensory integration and concurrent navigation strategies for spatial cognition in real and artificial organisms. , 2007, Journal of integrative neuroscience.

[9]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[10]  Patrick Pirim,et al.  An Integrated Neuromimetic Model of the Saccadic Eye Movements for the Psikharpax Robot , 2010, SAB.

[11]  Mehdi Khamassi,et al.  Combining Self-organizing Maps with Mixtures of Experts: Application to an Actor-Critic Model of Reinforcement Learning in the Basal Ganglia , 2006, SAB.

[12]  E. Gat.  On Three-Layer Architectures , 1997 .

[13]  Niko Wilbert,et al.  Modular Toolkit for Data Processing (MDP): A Python Data Processing Framework , 2008, Frontiers Neuroinformatics.

[14]  Jean-Arcady Meyer,et al.  Handbook of Robotics Chapter 61 : Biologically-inspired robots , 2007 .

[15]  E. Save,et al.  Coding for spatial goals in the prelimbic/infralimbic area of the rat frontal cortex. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[16]  Pierre Comon.  Independent component analysis - a new concept? signal processing , 1994 .

[17]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[18]  B. Schölkopf,et al.  Kernel Hebbian Algorithm for Iterative Kernel Principal Component Analysis , 2003 .

[19]  Thomas Dean,et al.  A Retrospective of the AAAI Robot Competitions , 1997, AI Mag..

[20]  Bernd Fritzke,et al.  A Growing Neural Gas Network Learns Topologies , 1994, NIPS.

[21]  N. White,et al.  Parallel Information Processing in the Dorsal Striatum: Relation to Hippocampal Function , 1999, The Journal of Neuroscience.

[22]  H. Yin,et al.  The role of the basal ganglia in habit formation , 2006, Nature Reviews Neuroscience.

[23]  Matthijs A. A. van der Meer,et al.  Theta Phase Precession in Rat Ventral Striatum Links Place and Reward Information , 2011, The Journal of Neuroscience.

[24]  M. Khamassi,et al.  Replay of rule-learning related neural patterns in the prefrontal cortex during sleep , 2009, Nature Neuroscience.

[25]  S. Killcross,et al.  Coordination of actions and habits in the medial prefrontal cortex of rats. , 2003, Cerebral cortex.

[26]  Ricardo Chavarriaga,et al.  A Computational Model of Parallel Navigation Systems in Rodents , 2005 .

[27]  K. Doya,et al.  Representation of Action-Specific Reward Values in the Striatum , 2005, Science.

[28]  Robin R. Murphy,et al.  Artificial intelligence and mobile robots: case studies of successful robot systems , 1998 .

[29]  E. Save,et al.  Hippocampal‐parietal cortical interactions in spatial cognition , 2000, Hippocampus.

[30]  P. Comon.  Independent Component Analysis , 1992 .

[31]  P. Dayan,et al.  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control , 2005, Nature Neuroscience.

[32]  Mehdi Khamassi,et al.  Complementary roles of the rat prefrontal cortex and striatum in reward-based learning and shifting navigation strategies. (Rôles complémentaires du cortex préfrontal et du striatum dans l'apprentissage et le changement de stratégies de navigation basées sur la récompense chez le rat) , 2007 .

[33]  Etienne Coutureau,et al.  A Role for Medial Prefrontal Dopaminergic Innervation in Instrumental Conditioning , 2009, The Journal of Neuroscience.

[34]  Reid G. Simmons,et al.  Robotic Systems Architectures and Programming , 2008, Springer Handbook of Robotics.

[35]  M. Packard,et al.  Differential effects of fornix and caudate nucleus lesions on two radial maze tasks: evidence for multiple memory systems , 1989, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[36]  Jean-Arcady Meyer,et al.  Biologically Inspired Robots , 2008, Springer Handbook of Robotics.

[37]  Maryam Yarmohamadi,et al.  Improvement of Robot Path Planning Using Particle Swarm Optimization in Dynamic Environments with Mobile Obstacles and Target , 2011 .

[38]  B. Knowlton,et al.  Contributions of striatal subregions to place and response learning. , 2004, Learning & memory.

[39]  R. Passingham.  The hippocampus as a cognitive map J. O'Keefe & L. Nadel, Oxford University Press, Oxford (1978). 570 pp., £25.00 , 1979, Neuroscience.

[40]  E. Miller,et al.  An integrative theory of prefrontal cortex function. , 2001, Annual review of neuroscience.

[41]  A. Graybiel.  The Basal Ganglia and Chunking of Action Repertoires , 1998, Neurobiology of Learning and Memory.

[42]  Keiji Tanaka,et al.  Medial prefrontal cell activity signaling prediction errors of action values , 2007, Nature Neuroscience.

[43]  Cyriel M. A. Pennartz,et al.  Learning-related changes in response patterns of prefrontal neurons during instrumental conditioning , 2003, Behavioural Brain Research.

[44]  Joel L. Davis,et al.  A Model of How the Basal Ganglia Generate and Use Neural Signals That Predict Reinforcement , 1994 .

[45]  S. N’guyen.  Mise au point du système vibrissal du robot-rat Psikharpax et contribution à la fusion de ses capacités visuelle, auditive et tactile , 2010 .

[46]  B. Knowlton,et al.  Learning and memory functions of the Basal Ganglia. , 2002, Annual review of neuroscience.

[47]  M. Jung,et al.  Dynamics of Population Code for Working Memory in the Prefrontal Cortex , 2003, Neuron.

[48]  J. D. McGaugh,et al.  Inactivation of Hippocampus or Caudate Nucleus with Lidocaine Differentially Affects Expression of Place and Response Learning , 1996, Neurobiology of Learning and Memory.

[49]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[50]  J. Mink.  THE BASAL GANGLIA: FOCUSED SELECTION AND INHIBITION OF COMPETING MOTOR PROGRAMS , 1996, Progress in Neurobiology.

[51]  Jeffrey L. Krichmar,et al.  Spatial navigation and causal analysis in a brain-based device modeling cortical-hippocampal interactions , 2007, Neuroinformatics.

[53]  Jean-Arcady Meyer,et al.  Phonotaxis behavior in the artificial rat Psikharpax , 2010 .

[54]  Angelo Arleo,et al.  Spatial Learning and Action Planning in a Prefrontal Cortical Network Model , 2011, PLoS Comput. Biol..

[55]  B. Balleine,et al.  Lesions of Medial Prefrontal Cortex Disrupt the Acquisition But Not the Expression of Goal-Directed Learning , 2005, The Journal of Neuroscience.

[56]  G. E. Alexander,et al.  Parallel organization of functionally segregated circuits linking basal ganglia and cortex. , 1986, Annual review of neuroscience.

[57]  Patrick Pirim,et al.  Tactile Texture Discrimination in the Robot-rat Psikharpax , 2010, BIOSIGNALS.

[58]  Michael E. Hasselmo,et al.  A Model of Prefrontal Cortical Mechanisms for Goal-directed Behavior , 2005, Journal of Cognitive Neuroscience.

[59]  Gordon Wyeth,et al.  Persistent Navigation and Mapping using a Biologically Inspired SLAM System , 2010, Int. J. Robotics Res..

[60]  Tamás Kiss,et al.  Episodes in Space: A Modeling Study of Hippocampal Place Representation , 2008, SAB.

[61]  P. Mahalanobis.  On the generalized distance in statistics , 1936 .

[62]  A. Parker.  Binocular depth perception and the cerebral cortex , 2007, Nature Reviews Neuroscience.

[63]  M. Roesch,et al.  Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards , 2007, Nature Neuroscience.

[64]  K. Doya.  Complementary roles of basal ganglia and cerebellum in learning and motor control , 2000, Current Opinion in Neurobiology.

[65]  M. Khamassi,et al.  Spatial decisions and neuronal activity in hippocampal projection zones in prefrontal cortex and striatum , 2008 .

[66]  J. MacQueen.  Some methods for classification and analysis of multivariate observations , 1967 .

[67]  N. Burgess.  Spatial Cognition and the Brain , 2008, Annals of the New York Academy of Sciences.

[68]  Bruno Poucet,et al.  Involvement of the rat prefrontal cortex in cognitive functions: A central role for the prelimbic area , 2000, Psychobiology.

[69]  Matthijs A. A. van der Meer,et al.  Integrating hippocampus and striatum in decision-making , 2007, Current Opinion in Neurobiology.

[70]  P. Dayan,et al.  Cortical substrates for exploratory decisions in humans , 2006, Nature.

[71]  A. Dickinson.  Actions and habits: the development of behavioural autonomy , 1985 .

[72]  Jean-Arcady Meyer,et al.  The Psikharpax project: towards building an artificial rat , 2005, Robotics Auton. Syst..

[73]  R. Morris.  Spatial Localization Does Not Require the Presence of Local Cues , 1981 .

[74]  Philippe Gaussier,et al.  Neurobiologically Inspired Mobile Robot Navigation and Planning , 2007, Frontiers in neurorobotics.

[75]  Erin L. Rich,et al.  Rat Prefrontal Cortical Neurons Selectively Code Strategy Switches , 2009, The Journal of Neuroscience.

[76]  Ricardo Chavarriaga,et al.  Path planning versus cue responding: a bio-inspired model of switching between navigation strategies , 2010, Biological Cybernetics.

[77]  Matthijs A. A. van der Meer,et al.  Ventral striatum: a critical look at models of learning and evaluation , 2011, Current Opinion in Neurobiology.

[78]  Alejandra Barrera,et al.  Biologically-inspired robot spatial cognition based on rat neurophysiological studies , 2008, Auton. Robots.

[79]  Ricardo Chavarriaga,et al.  Analyzing Interactions between Cue-Guided and Place-Based Navigation with a Computational Model of Action Selection: Influence of Sensory Cues and Training , 2010, SAB.

[80]  Rodrigo F. Salazar,et al.  NMDA lesions in the medial prefrontal cortex impair the ability to inhibit responses during reversal of a simple spatial discrimination , 2004, Behavioural Brain Research.

[81]  W. Kargo,et al.  Adaptation of Prefrontal Cortical Firing Patterns and Their Fidelity to Changes in Action–Reward Contingencies , 2007, The Journal of Neuroscience.

[82]  Sidney I. Wiener,et al.  Lesions of the medial shell of the nucleus accumbens impair rats in finding larger rewards, but spare reward-seeking behavior , 2000, Behavioural Brain Research.

[83]  D. Eilam,et al.  Home base behavior of rats (Rattus norvegicus) exploring a novel environment , 1989, Behavioural Brain Research.

[84]  Michael A. Arbib,et al.  Handbook of Robotics, Neurorobotics: From Vision to Action , 2008 .

[85]  G. Edelman,et al.  Retrospective and prospective responses arising in a modeled hippocampus during maze navigation by a brain-based device , 2007, Proceedings of the National Academy of Sciences.

[86]  Peter Dayan,et al.  Technical Note: Q-Learning , 2004, Machine Learning.

[87]  Philippe Gaussier,et al.  Autonomous vision-based navigation: Goal-oriented action planning by transient states prediction, cognitive map building, and sensory-motor learning , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[88]  R. Morris.  Developments of a water-maze procedure for studying spatial learning in the rat , 1984, Journal of Neuroscience Methods.

[89]  Jean-Arcady Meyer,et al.  BIOLOGICALLY BASED ARTIFICIAL NAVIGATION SYSTEMS: REVIEW AND PROSPECTS , 1997, Progress in Neurobiology.

[90]  Philippe Gaussier,et al.  A Hierarchy of Associations in Hippocampo-Cortical Systems: Cognitive Maps and Navigation Strategies , 2005, Neural Computation.

[91]  Amir Dezfouli,et al.  Speed/Accuracy Trade-Off between the Habitual and the Goal-Directed Processes , 2011, PLoS Comput. Biol..

[92]  Teuvo Kohonen,et al.  Self-Organizing Maps , 2010 .

[93]  R. Mishra,et al.  Self-Organization , 2021, Encyclopedic Dictionary of Archaeology.

[94]  Karl Pearson F.R.S..  LIII. On lines and planes of closest fit to systems of points in space , 1901 .

[95]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[96]  Angelo Arleo,et al.  Spatial cognition and neuro-mimetic navigation: a model of hippocampal place cell activity , 2000, Biological Cybernetics.