Motor primitive and sequence self-organization in a hierarchical recurrent neural network

This study describes how complex goal-directed behavior can be obtained through adaptation processes in a hierarchically organized recurrent neural network using a genetic algorithm (GA). Our experiments, using a simulated Khepera robot, showed that different types of dynamic structures self-organize in the lower and higher levels of the network for the purpose of achieving complex navigation tasks. The parametric bifurcation structures that appear in the lower level explain the mechanism of how behavior primitives are switched in a top-down way. In the higher level, a topologically ordered mapping of initial cell activation states to motor primitive sequences self-organizes by utilizing the initial sensitivity characteristics of non-linear dynamical systems. The biological plausibility of the model's essential principles is discussed.

[1]  Vaughan Vc rd,et al.  GROWTH AND DEVELOPMENT. , 1965, The Journal of the Arkansas Medical Society.

[2]  Robert A. Jacobs,et al.  Hierarchical Mixtures of Experts and the EM Algorithm , 1993, Neural Computation.

[3]  E. Bizzi,et al.  Linear combinations of primitives in vertebrate motor control. , 1994, Proceedings of the National Academy of Sciences of the United States of America.

[4]  Maja J. Mataric,et al.  Parametric primitives for motor representation and control , 2002, Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292).

[5]  Robert M. French,et al.  Using Semi-Distributed Representations to Overcome Catastrophic Forgetting in Connectionist Networks , 1991 .

[6]  Stefan Schaal,et al.  Is imitation learning the route to humanoid robots? , 1999, Trends in Cognitive Sciences.

[7]  Andrew Philippides,et al.  Temporally adaptive networks: Analysis of GasNet robot control. , 2002 .

[8]  J. Tanji,et al.  Integration of temporal order and object information in the monkey lateral prefrontal cortex. , 2004, Journal of neurophysiology.

[9]  Jun Tani,et al.  Learning to Generate Combinatorial Action Sequences Utilizing the Initial Sensitivity of Deterministic Dynamical Systems , 2003, IWANN.

[10]  P. Frank,et al.  Boston Studies in the Philosophy of Science , 1968 .

[11]  David E. Goldberg,et al.  The Design of Innovation: Lessons from and for Competent Genetic Algorithms , 2002 .

[12]  Chrystopher L. Nehaniv,et al.  Sensory-Motor Primitives as a Basis for Imitation: Linking Perception to Action and Biology to Robotics , 2002 .

[13]  S. Grossberg.  Studies of mind and brain : neural principles of learning, perception, development, cognition, and motor control , 1982 .

[14]  Dario Floreano,et al.  Levels of dynamics and adaptive behavior in evolutionary neural controllers , 2002 .

[15]  Francesco Mondada,et al.  Automatic creation of an autonomous agent: genetic evolution of a neural-network driven robot , 1994 .

[16]  Thomas G. Dietterich.  Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition , 1999, J. Artif. Intell. Res..

[17]  Tom Ziemke,et al.  Neuromodulation of Reactive Sensorimotor Mappings as a Short-Term Memory Mechanism in Delayed Response Tasks , 2002, Adapt. Behav..

[18]  Doina Precup,et al.  Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning , 1999, Artif. Intell..

[19]  Jeffrey L. Krichmar,et al.  Evolutionary robotics: The biology, intelligence, and technology of self-organizing machines , 2001, Complex..

[20]  Inman Harvey,et al.  An Evolutionary Ecological Approach to the Study of Learning Behavior Using a Robot-Based Model , 2002, Adapt. Behav..

[21]  T. Michael Knasel,et al.  Robotics and autonomous systems , 1988, Robotics Auton. Syst..

[22]  Dorothea Heiss-Czedik,et al.  An Introduction to Genetic Algorithms. , 1997, Artificial Life.

[23]  Jesper Blynel,et al.  Evolving Reinforcement Learning-Like Abilities for Robots , 2003, ICES.

[24]  Stephen Grossberg,et al.  Studies of mind and brain , 1982 .

[25]  Randall D. Beer,et al.  Evolving Dynamical Neural Networks for Adaptive Behavior , 1992, Adapt. Behav..

[26]  Richard S. Sutton,et al.  Roles of Macro-Actions in Accelerating Reinforcement Learning , 1998 .

[27]  Aude Billard,et al.  From Animals to Animats , 2004 .

[28]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[29]  K. Dautenhahn,et al.  Imitation in Animals and Artifacts , 2002 .

[30]  Inman Harvey,et al.  Evolutionary robotics: the Sussex approach , 1997, Robotics Auton. Syst..

[31]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[32]  Geoffrey E. Hinton,et al.  Adaptive Mixtures of Local Experts , 1991, Neural Computation.

[33]  Mitsuo Kawato,et al.  Multiple Model-Based Reinforcement Learning , 2002, Neural Computation.

[34]  Randall D. Beer,et al.  Sequential Behavior and Learning in Evolved Dynamical Neural Networks , 1994, Adapt. Behav..

[35]  Richard Fikes,et al.  Learning and Executing Generalized Robot Plans , 1993, Artif. Intell..

[36]  Jianqing Fan,et al.  Estimation of conditional densities and sensitivity measures in nonlinear dynamical systems , 1996 .

[37]  Bridget Hallam.  From animals to animats 7 : proceedings of the seventh International Conference on Simulation of Adaptive Behavior , 2002 .

[38]  I Kupfermann,et al.  An identified neuron (CPR) evokes neuronal responses reflecting food arousal in Aplysia. , 1990, Science.

[39]  Jun Tanji,et al.  Role for supplementary motor area cells in planning several movements ahead , 1994, Nature.

[40]  Michael McCloskey,et al.  Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem , 1989 .

[41]  W. J. Heitler,et al.  Fifty years of a command neuron: the neurobiology of escape behavior in the crayfish , 1999, Trends in Neurosciences.

[42]  Eytan Ruppin,et al.  Spontaneous Evolution of Command Neurons, Place Cells and Memory Mechanisms in Autonomous Agents , 1999, ECAL.

[43]  Jun Nakanishi,et al.  Learning Attractor Landscapes for Learning Motor Primitives , 2002, NIPS.

[44]  Stefano Nolfi,et al.  Learning to perceive the world as articulated: an approach for hierarchical learning in sensory-motor systems , 1998, Neural Networks.

[45]  K. Kaneko,et al.  How fast elements can affect slow dynamics , 2001, nlin/0108038.

[46]  P J Webros.  BACKPROPAGATION THROUGH TIME: WHAT IT DOES AND HOW TO DO IT , 1990 .

[47]  Bertrand Mesot,et al.  SOS++: finding smart behaviors using learning and evolution , 2002 .

[48]  Doina Precup,et al.  Multi-time Models for Temporally Abstract Planning , 1997, NIPS.

[49]  F. A. Mussa-lvaldi,et al.  Convergent force fields organized in the frog's spinal cord , 1993, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[50]  Stefano Nolfi,et al.  Evolving robots able to self-localize in the environment: the importance of viewing cognition as the result of processes occurring at different time-scales , 2002 .

[51]  M. Kawato,et al.  A hierarchical neural-network model for control and learning of voluntary movement , 2004, Biological Cybernetics.

[52]  Michael A. Arbib,et al.  Perceptual Structures and Distributed Motor Control , 1981 .

[53]  Reza Shadmehr,et al.  Learning of action through adaptive combination of motor primitives , 2000, Nature.

[54]  Jun Tani,et al.  Learning to generate articulated behavior through the bottom-up and the top-down interaction processes , 2003, Neural Networks.

[55]  Ron Sun,et al.  From implicit skills to explicit knowledge: a bottom-up model of skill learning , 2001, Cogn. Sci..

[56]  Hava T. Siegelmann,et al.  On the Computational Power of Neural Nets , 1995, J. Comput. Syst. Sci..

[57]  O. Hikosaka,et al.  Chunking during human visuomotor sequence learning , 2003, Experimental Brain Research.

[58]  Andrew G. Barto,et al.  Accelerating Reinforcement Learning through the Discovery of Useful Subgoals , 2001 .

[59]  D M Wolpert,et al.  Multiple paired forward and inverse models for motor control , 1998, Neural Networks.

[60]  Stuart J. Russell,et al.  Reinforcement Learning with Hierarchies of Machines , 1997, NIPS.