Integrative Learning between Language and Action: A Neuro-Robotics Experiment

This paper introduces a model for associative learning combining both linguistic and behavior modalities. The model consists of language and behavior modules both implemented by a hierarchical dynamic network model and interacting densely through hub-like neurons, the so-called parametric biases (PB). By implementing this model for a humanoid robot with the task of manipulating multiple objects, the robot was tutored to associate sentences of two different grammatical types with corresponding sensory-motor schemata. The first type was a verb followed by an objective noun such as "hold red" or "hit blue"; the second was a verb followed by an objective noun and further followed by an adverb phrase such as "Put red on blue". Our analysis of the results of a learning experiment showed that two clusters corresponding to these two types of grammatical sentences appear in the PB activity space, such that a specific micro structure is organized for each cluster.