Ab Initio Particle-based Object Manipulation

Particle-based Object Manipulation (PROMPT) is a new method for robot manipulation of novel objects, without prior object models or pre-training on a large object data set. The key element of PROMPT is a particle-based object representation, in which each particle represents a point in an object, the local geometric, physical, and other features of the point, and also its relation with other particles. The particle representation connects visual perception with robot control. Like data-driven methods, PROMPT infers the object representation online in real time from the visual sensor. Like model-based methods, PROMPT leverages the particle representation to reason about the object’s geometry and dynamics, and choose suitable manipulation actions accordingly. PROMPT thus combines the strengths of model-based and data-driven methods. We show empirically that PROMPT successfully handles a variety of everyday objects, some of which are transparent. It handles various manipulation tasks, including grasping, pushing, etc. . Our experiments also show that PROMPT outperforms a stateof-the-art data-driven grasping method on everyday household objects, even though it does not use any offline training data. The code and a demonstration video are available online