A Cognitive System for Human Manipulation Action Understanding

This paper describes the architecture of a cognitive system that interprets human manipulation actions from perceptual information (image and depth data) and consists of perceptual modules and reasoning modules that are in interaction with each other. The contributions of our work are along two core problems at the heart of action understanding: a.) the grounding of relevant information about actions in perception (the perception action integration problem), and b.) the organization of perceptual and high-level symbolic information for interpreting the actions (the sequencing problem). At the high level, actions are represented with the Manipulation Action Grammar, a context-free grammar and associated parsing algorithms, which organizes actions as a sequence of sub-events. Each sub-event is described by the hand, movements and the objects and tools involved, and the relevant information about these quantities is obtained from biological-inspired perception modules. These modules track the hands and objects and recognize the hand grasp, objects and actions using attention, segmentation, and feature description. Experiments on a new dataset of manipulation actions show that our system can successfully extract the relevant visual information and semantic representation. This representation could further be used by the cognitive agent for reasoning, prediction, and planning.