A biologically inspired meta-control navigation system for the Psikharpax rat robot

A biologically inspired navigation system for the mobile rat-like robot named Psikharpax is presented, allowing for self-localization and autonomous navigation in an initially unknown environment. The ability of parts of the model (e.g. the strategy selection mechanism) to reproduce rat behavioral data in various maze tasks has been validated before in simulations. But the capacity of the model to work on a real robot platform had not been tested. This paper presents our work on the implementation on the Psikharpax robot of two independent navigation strategies (a place-based planning strategy and a cue-guided taxon strategy) and a strategy selection meta-controller. We show how our robot can memorize which was the optimal strategy in each situation, by means of a reinforcement learning algorithm. Moreover, a context detector enables the controller to quickly adapt to changes in the environment—recognized as new contexts—and to restore previously acquired strategy preferences when a previously experienced context is recognized. This produces adaptivity closer to rat behavioral performance and constitutes a computational proposition of the role of the rat prefrontal cortex in strategy shifting. Moreover, such a brain-inspired meta-controller may provide an advancement for learning architectures in robotics.

[1]  Matthijs A. A. van der Meer,et al.  Ventral striatum: a critical look at models of learning and evaluation , 2011, Current Opinion in Neurobiology.

[2]  Angelo Arleo,et al.  Spatial Learning and Action Planning in a Prefrontal Cortical Network Model , 2011, PLoS Comput. Biol..

[3]  Amir Dezfouli,et al.  Speed/Accuracy Trade-Off between the Habitual and the Goal-Directed Processes , 2011, PLoS Comput. Biol..

[4]  Matthijs A. A. van der Meer,et al.  Theta Phase Precession in Rat Ventral Striatum Links Place and Reward Information , 2011, The Journal of Neuroscience.

[5]  Ricardo Chavarriaga,et al.  Path planning versus cue responding: a bio-inspired model of switching between navigation strategies , 2010, Biological Cybernetics.

[6]  Patrick Pirim,et al.  An Integrated Neuromimetic Model of the Saccadic Eye Movements for the Psikharpax Robot , 2010, SAB.

[7]  Ricardo Chavarriaga,et al.  Analyzing Interactions between Cue-Guided and Place-Based Navigation with a Computational Model of Action Selection: Influence of Sensory Cues and Training , 2010, SAB.

[8]  Gordon Wyeth,et al.  Persistent Navigation and Mapping using a Biologically Inspired SLAM System , 2010, Int. J. Robotics Res..

[9]  Teuvo Kohonen,et al.  Self-Organizing Maps , 2010 .

[10]  M. Khamassi,et al.  Replay of rule-learning related neural patterns in the prefrontal cortex during sleep , 2009, Nature Neuroscience.

[11]  Erin L. Rich,et al.  Rat Prefrontal Cortical Neurons Selectively Code Strategy Switches , 2009, The Journal of Neuroscience.

[12]  Etienne Coutureau,et al.  A Role for Medial Prefrontal Dopaminergic Innervation in Instrumental Conditioning , 2009, The Journal of Neuroscience.

[13]  Philippe Gaussier,et al.  Autonomous vision-based navigation: Goal-oriented action planning by transient states prediction, cognitive map building, and sensory-motor learning , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[14]  Michael A. Arbib,et al.  Handbook of Robotics, Neurorobotics: From Vision to Action , 2008 .

[15]  Alejandra Barrera,et al.  Biologically-inspired robot spatial cognition based on rat neurophysiological studies , 2008, Auton. Robots.

[16]  Tamás Kiss,et al.  Episodes in Space: A Modeling Study of Hippocampal Place Representation , 2008, SAB.

[17]  N. Burgess.  Spatial Cognition and the Brain , 2008, Annals of the New York Academy of Sciences.

[18]  Matthijs A. A. van der Meer,et al.  Integrating hippocampus and striatum in decision-making , 2007, Current Opinion in Neurobiology.

[19]  M. Roesch,et al.  Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards , 2007, Nature Neuroscience.

[20]  R. Pfeifer,et al.  Self-Organization, Embodiment, and Biologically Inspired Robotics , 2007, Science.

[21]  Adam Johnson,et al.  Neural Ensembles in CA3 Transiently Encode Paths Forward of the Animal at a Decision Point , 2007, The Journal of Neuroscience.

[22]  Mehdi Khamassi,et al.  Complementary roles of the rat prefrontal cortex and striatum in reward-based learning and shifting navigation strategies. (Rôles complémentaires du cortex préfrontal et du striatum dans l'apprentissage et le changement de stratégies de navigation basées sur la récompense chez le rat) , 2007 .

[23]  Philippe Gaussier,et al.  Neurobiologically Inspired Mobile Robot Navigation and Planning , 2007, Frontiers in neurorobotics.

[24]  Angelo Arleo,et al.  Multimodal sensory integration and concurrent navigation strategies for spatial cognition in real and artificial organisms. , 2007, Journal of integrative neuroscience.

[25]  Keiji Tanaka,et al.  Medial prefrontal cell activity signaling prediction errors of action values , 2007, Nature Neuroscience.

[26]  A. Parker.  Binocular depth perception and the cerebral cortex , 2007, Nature Reviews Neuroscience.

[27]  W. Kargo,et al.  Adaptation of Prefrontal Cortical Firing Patterns and Their Fidelity to Changes in Action–Reward Contingencies , 2007, The Journal of Neuroscience.

[28]  G. Edelman,et al.  Retrospective and prospective responses arising in a modeled hippocampus during maze navigation by a brain-based device , 2007, Proceedings of the National Academy of Sciences.

[29]  Mehdi Khamassi,et al.  Combining Self-organizing Maps with Mixtures of Experts: Application to an Actor-Critic Model of Reinforcement Learning in the Basal Ganglia , 2006, SAB.

[30]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[31]  P. Dayan,et al.  Cortical substrates for exploratory decisions in humans , 2006, Nature.

[32]  H. Yin,et al.  The role of the basal ganglia in habit formation , 2006, Nature Reviews Neuroscience.

[33]  P. Dayan,et al.  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control , 2005, Nature Neuroscience.

[34]  K. Doya,et al.  Representation of Action-Specific Reward Values in the Striatum , 2005, Science.

[35]  B. Balleine,et al.  Lesions of Medial Prefrontal Cortex Disrupt the Acquisition But Not the Expression of Goal-Directed Learning , 2005, The Journal of Neuroscience.

[36]  Michael E. Hasselmo,et al.  A Model of Prefrontal Cortical Mechanisms for Goal-directed Behavior , 2005, Journal of Cognitive Neuroscience.

[37]  Philippe Gaussier,et al.  A Hierarchy of Associations in Hippocampo-Cortical Systems: Cognitive Maps and Navigation Strategies , 2005, Neural Computation.

[38]  Jean-Arcady Meyer,et al.  The Psikharpax project: towards building an artificial rat , 2005, Robotics Auton. Syst..

[39]  E. Save,et al.  Coding for spatial goals in the prelimbic/infralimbic area of the rat frontal cortex. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[40]  Rodrigo F. Salazar,et al.  NMDA lesions in the medial prefrontal cortex impair the ability to inhibit responses during reversal of a simple spatial discrimination , 2004, Behavioural Brain Research.

[41]  B. Knowlton,et al.  Contributions of striatal subregions to place and response learning. , 2004, Learning & memory.

[42]  Cyriel M. A. Pennartz,et al.  Learning-related changes in response patterns of prefrontal neurons during instrumental conditioning , 2003, Behavioural Brain Research.

[43]  B. Kolb,et al.  Do rats have a prefrontal cortex? , 2003, Behavioural Brain Research.

[44]  M. Jung,et al.  Dynamics of Population Code for Working Memory in the Prefrontal Cortex , 2003, Neuron.

[45]  B. Schölkopf,et al.  Kernel Hebbian Algorithm for Iterative Kernel Principal Component Analysis , 2003 .

[46]  S. Killcross,et al.  Coordination of actions and habits in the medial prefrontal cortex of rats. , 2003, Cerebral cortex.

[47]  M. Zugaro,et al.  Lesions of the medial shell of the nucleus accumbens impair rats in finding larger rewards, but spare reward-seeking behavior , 2000, Behavioural Brain Research.

[48]  K. Doya.  Complementary roles of basal ganglia and cerebellum in learning and motor control , 2000, Current Opinion in Neurobiology.

[49]  Angelo Arleo,et al.  Spatial cognition and neuro-mimetic navigation: a model of hippocampal place cell activity , 2000, Biological Cybernetics.

[50]  V. Brown,et al.  Medial Frontal Cortex Mediates Perceptual Attentional Set Shifting in the Rat , 2000, The Journal of Neuroscience.

[51]  Bruno Poucet,et al.  Involvement of the rat prefrontal cortex in cognitive functions: A central role for the prelimbic area , 2000, Psychobiology.

[52]  R. Kesner,et al.  Involvement of the Prelimbic–Infralimbic Areas of the Rodent Prefrontal Cortex in Behavioral Flexibility for Place and Response Learning , 1999, The Journal of Neuroscience.

[53]  N. White,et al.  Parallel Information Processing in the Dorsal Striatum: Relation to Hippocampal Function , 1999, The Journal of Neuroscience.

[54]  John M. Pearce,et al.  Hippocampal lesions disrupt navigation based on cognitive maps but not heading vectors , 1998, Nature.

[55]  A. Graybiel.  The Basal Ganglia and Chunking of Action Repertoires , 1998, Neurobiology of Learning and Memory.

[56]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[57]  E. Gat.  On Three-Layer Architectures , 1997 .

[58]  Robin R. Murphy,et al.  Artificial intelligence and mobile robots: case studies of successful robot systems , 1998 .

[59]  Jean-Arcady Meyer,et al.  BIOLOGICALLY BASED ARTIFICIAL NAVIGATION SYSTEMS: REVIEW AND PROSPECTS , 1997, Progress in Neurobiology.

[60]  Thomas Dean,et al.  A Retrospective of the AAAI Robot Competitions , 1997, AI Mag..

[61]  J. Mink.  THE BASAL GANGLIA: FOCUSED SELECTION AND INHIBITION OF COMPETING MOTOR PROGRAMS , 1996, Progress in Neurobiology.

[62]  J. D. McGaugh,et al.  Inactivation of Hippocampus or Caudate Nucleus with Lidocaine Differentially Affects Expression of Place and Response Learning , 1996, Neurobiology of Learning and Memory.

[63]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[64]  P. Dayan,et al.  Q-learning , 1992, Machine Learning.

[65]  D. Eilam,et al.  Home base behavior of rats (Rattus norvegicus) exploring a novel environment , 1989, Behavioural Brain Research.

[66]  M. Packard,et al.  Differential effects of fornix and caudate nucleus lesions on two radial maze tasks: evidence for multiple memory systems , 1989, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[67]  A. Dickinson.  Actions and habits: the development of behavioural autonomy , 1985 .

[68]  R. Morris.  Developments of a water-maze procedure for studying spatial learning in the rat , 1984, Journal of Neuroscience Methods.

[69]  R. Morris.  Spatial Localization Does Not Require the Presence of Local Cues , 1981 .

[70]  M. Eckardt.  The Hippocampus as a Cognitive Map , 1980 .

[71]  L. Nadel,et al.  The Hippocampus as a Cognitive Map , 1978 .

[72]  Stephen J. Garland,et al.  Algorithm 97: Shortest path , 1962, Commun. ACM.

[73]  Karl Pearson F.R.S..  LIII. On lines and planes of closest fit to systems of points in space , 1901 .

[74]  Maryam Yarmohamadi,et al.  Improvement of Robot Path Planning Using Particle Swarm Optimization in Dynamic Environments with Mobile Obstacles and Target , 2011 .

[75]  S. N’guyen.  Mise au point du système vibrissal du robot-rat Psikharpax et contribution à la fusion de ses capacités visuelle, auditive et tactile , 2010 .

[76]  Patrick Pirim,et al.  Tactile Texture Discrimination in the Robot-rat Psikharpax , 2010, BIOSIGNALS.

[77]  Jean-Arcady Meyer,et al.  Phonotaxis behavior in the artificial rat Psikharpax , 2010 .

[78]  Seungjin Choi,et al.  Independent Component Analysis , 2009, Handbook of Natural Computing.

[79]  M. Khamassi,et al.  Spatial decisions and neuronal activity in hippocampal projection zones in prefrontal cortex and striatum , 2008 .

[80]  Jean-Arcady Meyer,et al.  Biologically Inspired Robots , 2008, Springer Handbook of Robotics.

[81]  Reid G. Simmons,et al.  Robotic Systems Architectures and Programming , 2008, Springer Handbook of Robotics, 2nd Ed..

[82]  Florent Lamiraux,et al.  Motion Planning and Obstacle Avoidance , 2008, Springer Handbook of Robotics.

[83]  W. Gerstner,et al.  A computational model of parallel navigation systems in rodents , 2007, Neuroinformatics.

[84]  Agnès Guillot,et al.  Handbook of Robotics Chapter 61: Biologically-inspired Robots , 2007 .

[85]  P. Read Montague,et al.  Reinforcement Learning: An Introduction , 2005, IEEE Transactions on Neural Networks.

[86]  P. Dayan,et al.  Technical Note: Q-Learning , 2004, Machine Learning.

[87]  B. Knowlton,et al.  Learning and memory functions of the Basal Ganglia. , 2002, Annual review of neuroscience.

[88]  E. Miller,et al.  An integrative theory of prefrontal cortex function. , 2001, Annual review of neuroscience.

[89]  E. Save,et al.  Hippocampal‐parietal cortical interactions in spatial cognition , 2000, Hippocampus.

[90]  A. Redish.  Beyond the Cognitive Map: From Place Cells to Episodic Memory , 1999 .

[91]  Bernd Fritzke,et al.  A Growing Neural Gas Network Learns Topologies , 1994, NIPS.

[92]  Joel L. Davis,et al.  A Model of How the Basal Ganglia Generate and Use Neural Signals That Predict Reinforcement , 1994 .

[93]  G. E. Alexander,et al.  Parallel organization of functionally segregated circuits linking basal ganglia and cortex. , 1986, Annual review of neuroscience.

[94]  J. MacQueen.  Some methods for classification and analysis of multivariate observations , 1967 .

[95]  P. Mahalanobis.  On the generalized distance in statistics , 1936 .

[96]  G. Edelman,et al.  Spatial navigation and causal analysis in a brain-based device modeling cortical-hippocampal interactions , 2007, Neuroinformatics.

[97]  Niko Wilbert,et al.  Modular Toolkit for Data Processing (MDP): A Python Data Processing Framework , 2008, Frontiers Neuroinformatics.