Imitation learning with hierarchical actions

Imitation is a powerful mechanism for rapidly learning new skills through observation of a mentor. Developmental studies indicate that children often perform goal-based imitation rather than mimicking a mentor's actual action trajectories. Further, imitation, and human behavior in general, appear to be based on a hierarchy of actions, with higher-level actions composed of sequences of lower-level actions. In this paper, we propose a new model for goal-based imitation that exploits action hierarchies for fast learning of new skills. As in human imitation, learning relies only on sample trajectories of mentor states. Unlike apprenticeship or inverse reinforcement learning, the model does not require that mentor actions be given. We present results from a large-scale grid world task that is modeled after a puzzle box task used in developmental studies for investigating hierarchical imitation in children. We show that the proposed model rapidly learns to combine a given set of hierarchical actions to achieve the subgoals necessary to reach a desired goal state. Our results demonstrate that hierarchical imitation can yield significant speed-up in learning, especially in large state spaces, compared to learning without a mentor or without an action hierarchy.