Grasping Affordances: Learning to Connet vission to Hand Action

When presented with an object to be manipulated, a robot must identify the available forms of interaction with the object. How might an agent automatically acquire this mapping from visual description of the object to manipulation action? In this chapter, we describe two components of an algorithm that enable an agent to learn a grasping-oriented representation by observing an object being manipulated by a human teacher. The first component uses the sequence of image/object pose tuples to acquire a model of the object’s appearance as a function of the viewing angle. We identify visual features that are robustly observable over a range of similar viewing angles, but that are also discriminative of the set of viewing angles. Given a novel image, the algorithm can then estimate the angle from which the object is being viewed. The second component of the algorithm clusters the sequence of observed hand postures into the functionally distinct ways that the object may be grasped. Experimental results demonstrate the feasibility of extracting a compact set of canonical grasps from this experience. Each of these canonical grasps can then be used to parameterize a reach controller that brings the robot hand into a specific spatial relationship with the object. Charles de Granville, e-mail: chazz184@gmail.com Di Wang, e-mail: di@cs.ou.edu Joshua Southerland, e-mail: Joshua.B.Southerland-1@ou.edu Andrew H. Fagg, e-mail: fagg@cs.ou.edu Symbiotic Computing Laboratory School of Computer Science University of Oklahoma Norman, OK Robert Platt, Jr., e-mail: robert.platt-1@nasa.gov Dexterous Robotics Laboratory Johnson Space Center, NASA Houston, TX