Hand action perception for robot programming

This paper presents a general and robust approach to hand action perception for automatic robot programming using depth image sequences. The human instructor must simply demonstrate an assembly task in front of a vision system in the human world; no dataglove or special markings are necessary. The recorded image sequences are used to recover a depth image sequence for model-based human hand and object tracking to form the perceptual data stream. The data stream is then segmented and interpreted for generating a task sequence which describes the human hand action and the relationship between the manipulated object and the hand. The task sequence might be composed of a series of subtasks and each subtask involves four phases: approaching, pre-manipulating, manipulating and departing. In this paper we also discuss a robot system that replicates the observed task and automatically validates the replication results in the robot world.