Inferring Generalized Pick-and-Place Tasks from Pointing Gestures

In this paper, we describe how to simplify the instruction of robots to perform pick-and-place tasks. This is achieved by inferring additional information about the environment, locations in it, and the used object, based on semantic knowledge. We use the Kinect RGB-D camera mounted on a PR2 robot and the human tracking system OpenNI that allows the interpretation of simple human gestures. Our application builds enables virtual movements of objects freely in space and the use of an ontology for reasoning about the plausibility of the tasks specified by releasing objects in certain locations. Our experiments show how relatively complex object manipulation tasks can be specified as simply as “drag-and-drop”.