Data-Driven Grasp Synthesis—A Survey

We review the work on data-driven grasp synthesis and the methodologies for sampling and ranking candidate grasps. We divide the approaches into three groups based on whether they synthesize grasps for known, familiar, or unknown objects. This structure allows us to identify common object representations and perceptual processes that facilitate the employed data-driven grasp synthesis technique. In the case of known objects, we concentrate on the approaches that are based on object recognition and pose estimation. In the case of familiar objects, the techniques use some form of a similarity matching to a set of previously encountered objects. Finally, for the approaches dealing with unknown objects, the core part is the extraction of specific features that are indicative of good grasps. Our survey provides an overview of the different methodologies and discusses open problems in the area of robot grasping. We also draw a parallel to the classical approaches that rely on analytic formulations.

[1]  Wayne D. Gray,et al.  Basic objects in natural categories , 1976, Cognitive Psychology.

[2]  Vincent Dupourqué,et al.  A robot operating system , 1984, ICRA.

[3]  Van-Duc Nguyen,et al.  Constructing force-closure grasps in 3D , 1987, Proceedings. 1987 IEEE International Conference on Robotics and Automation.

[4]  Van-Duc Nguyen,et al.  Constructing Force- Closure Grasps , 1988, Int. J. Robotics Res..

[5]  M. Goodale,et al.  Separate visual pathways for perception and action , 1992, Trends in Neurosciences.

[6]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  John F. Canny,et al.  Planning optimal grasps , 1992, Proceedings 1992 IEEE International Conference on Robotics and Automation.

[8]  S. Shankar Sastry,et al.  A mathematical introduction to robotics manipulation , 1994 .

[9]  Richard M. Murray,et al.  A Mathematical Introduction to Robotic Manipulation , 1994 .

[10]  Shimon Edelman,et al.  Learning to grasp using visual information , 1996, Proceedings of IEEE International Conference on Robotics and Automation.

[11]  Karun B. Shimoga,et al.  Robot Grasp Synthesis Algorithms: A Survey , 1996, Int. J. Robotics Res..

[12]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[13]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[14]  Henrik I. Christensen,et al.  Automatic grasp planning using shape primitives , 2003, 2003 IEEE International Conference on Robotics and Automation (Cat. No.03CH37422).

[15]  Gerd Hirzinger,et al.  Grasping the dice by dicing the grasp , 2003, Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No.03CH37453).

[16]  Giorgio Metta,et al.  Better Vision through Manipulation , 2003, Adapt. Behav..

[17]  Antonio Morales,et al.  Using Experience for Assessing Grasp Reliability , 2004, Int. J. Humanoid Robotics.

[18]  Gabriela Csurka,et al.  Visual categorization with bags of keypoints , 2002, eccv 2004.

[19]  Peter K. Allen,et al.  An SVM learning approach to robotic grasping , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[20]  Peter K. Allen,et al.  Graspit! A versatile simulator for robotic grasping , 2004, IEEE Robotics & Automation Magazine.

[21]  U. Castiello The neuroscience of grasping , 2005, Nature Reviews Neuroscience.

[22]  Pietro Perona,et al.  A Bayesian hierarchical model for learning natural scene categories , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[23]  Ying Li,et al.  A shape matching algorithm for synthesizing humanlike enveloping grasps , 2005, 5th IEEE-RAS International Conference on Humanoid Robots, 2005..

[24]  Tamim Asfour,et al.  Integrated Grasp Planning and Visual Object Localization For a Humanoid Robot with Five-Fingered Hands , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[25]  Bernt Schiele,et al.  An Implicit Shape Model for Combined Object Categorization and Segmentation , 2006, Toward Category-Level Object Recognition.

[26]  Cordelia Schmid,et al.  Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[27]  J. Culham,et al.  The role of parietal cortex in visuomotor control: What have we learned from neuroimaging? , 2006, Neuropsychologia.

[28]  Antonio Morales,et al.  Vision-based three-finger grasp synthesis constrained by hand geometry , 2006, Robotics Auton. Syst..

[29]  Antonio Criminisi,et al.  TextonBoost: Joint Appearance, Shape and Context Modeling for Multi-class Object Recognition and Segmentation , 2006, ECCV.

[30]  Dmitry Berenson,et al.  Grasp planning in complex scenes , 2007, 2007 7th IEEE-RAS International Conference on Humanoid Robots.

[31]  Danica Kragic,et al.  Learning and Evaluation of the Approach Vector for Automatic Grasp Generation and Planning , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[32]  Massimo Bergamasco,et al.  Haptic Simulation, Perception and Manipulation of Deformable Objects , 2007, Eurographics.

[33]  Tamim Asfour,et al.  Stereo-based 6D object localization for grasping with humanoid robot systems , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[34]  Peter K. Allen,et al.  Grasp Planning via Decomposition Trees , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[35]  Frédéric Jurie,et al.  Groups of Adjacent Contour Segments for Object Detection , 2008, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  Gary M. Bone,et al.  Automated modeling and robotic grasping of unknown three-dimensional objects , 2008, 2008 IEEE International Conference on Robotics and Automation.

[37]  Manuel Lopes,et al.  Learning Object Affordances: From Sensory--Motor Coordination to Imitation , 2008, IEEE Transactions on Robotics.

[38]  Antonio Morales,et al.  Vision-based grasp planning of 3D objects by extending 2D contour based algorithms , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[39]  Danica Kragic,et al.  Birth of the Object: Detection of Objectness and Extraction of Object Shape through Object-Action complexes , 2008, Int. J. Humanoid Robotics.

[40]  Bernt Schiele,et al.  Functional Object Class Detection Based on Learned Affordance Cues , 2008, ICVS.

[41]  Claire Dune,et al.  Active rough shape estimation of unknown objects , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[42]  Lawson L. S. Wong,et al.  Learning Grasp Strategies with Partial Shape Information , 2008, AAAI.

[43]  Anis Sahbani,et al.  Handling Objects by Their Handles , 2008 .

[44]  Jing Xiao,et al.  Efficient and effective grasping of novel objects through learning and adapting a knowledge base , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[45]  Ashutosh Saxena,et al.  Robotic Grasping of Novel Objects using Vision , 2008, Int. J. Robotics Res..

[46]  Mario Richtsfeld,et al.  Grasping of Unknown Objects from a Table Top , 2008 .

[47]  Danica Kragic,et al.  Selection of robot pre-grasps using box-based shape approximation , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[48]  James J. Kuffner,et al.  OpenRAVE: A Planning Architecture for Autonomous Robotics , 2008 .

[49]  Tamim Asfour,et al.  Toward humanoid manipulation in human-centred environments , 2008, Robotics Auton. Syst..

[50]  Nicholas Roy,et al.  Probabilistic Models of Object Geometry for Grasp Planning , 2008, Robotics: Science and Systems.

[51]  Zoltan-Csaba Marton,et al.  Probabilistic categorization of kitchen objects in table settings with a composite sensor , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[52]  Javier Felip,et al.  Robust sensor-based grasp primitive for a three-finger robot hand , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[53]  N. Kruger,et al.  Learning object-specific grasp affordance densities , 2009, 2009 IEEE 8th International Conference on Development and Learning.

[54]  Danica Kragic,et al.  Grasping known objects with humanoid robots: A box-based approach , 2009, 2009 International Conference on Advanced Robotics.

[55]  Eris Chinellato,et al.  The neuroscience of vision-based grasping: a functional review for computational modeling and bio-inspired robotics. , 2009, Journal of integrative neuroscience.

[56]  Danica Kragic,et al.  Grasp recognition and mapping on humanoid robots , 2009, 2009 9th IEEE-RAS International Conference on Humanoid Robots.

[57]  Ashutosh Saxena,et al.  Reactive grasping using optical proximity sensors , 2009, 2009 IEEE International Conference on Robotics and Automation.

[58]  Danica Kragic,et al.  Demonstration-based learning and control for automatic grasping , 2009, Intell. Serv. Robotics.

[59]  Justus H. Piater,et al.  A Probabilistic Framework for 3D Visual Object Representation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[60]  Siddhartha S. Srinivasa,et al.  Object recognition and full pose registration from a single image for robotic manipulation , 2009, 2009 IEEE International Conference on Robotics and Automation.

[61]  Nico Blodow,et al.  Fast Point Feature Histograms (FPFH) for 3D registration , 2009, 2009 IEEE International Conference on Robotics and Automation.

[62]  Oliver Brock,et al.  Interactive segmentation for manipulation in unstructured environments , 2009, 2009 IEEE International Conference on Robotics and Automation.

[63]  Máximo A. Roa,et al.  Computation of Independent Contact Regions for Grasping 3-D Objects , 2009, IEEE Transactions on Robotics.

[64]  Matei T. Ciocarlie,et al.  The Columbia grasp database , 2009, 2009 IEEE International Conference on Robotics and Automation.

[65]  Danica Kragic,et al.  Modeling and evaluation of human-to-robot mapping of grasps , 2009, 2009 International Conference on Advanced Robotics.

[66]  Matei T. Ciocarlie,et al.  Hand Posture Subspaces for Dexterous Robotic Grasping , 2009, Int. J. Robotics Res..

[67]  Michael Beetz,et al.  Detecting and segmenting objects for mobile manipulation , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[68]  Niklas Bergström,et al.  Integration of Visual Cues for Robotic Grasping , 2009, ICVS.

[69]  Matei T. Ciocarlie,et al.  Contact-reactive grasping of objects with partial shape information , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[70]  Justus H. Piater,et al.  Refining grasp affordance models by experience , 2010, 2010 IEEE International Conference on Robotics and Automation.

[71]  Oliver Kroemer,et al.  Combining active learning and reactive control for robot grasping , 2010, Robotics Auton. Syst..

[72]  Darius Burschka,et al.  An Efficient RANSAC for 3D Object Recognition in Noisy and Occluded Scenes , 2010, ACCV.

[73]  Michael Beetz,et al.  Robotic grasping of unmodeled objects using time-of-flight range data and finger torque information , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[74]  Matei T. Ciocarlie,et al.  Towards Reliable Grasping and Manipulation in Household Environments , 2010, ISER.

[75]  Takeo Kanade,et al.  Automated Construction of Robotic Manipulation Programs , 2010 .

[76]  Quoc V. Le,et al.  Grasping novel objects with depth segmentation , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[77]  Tamim Asfour,et al.  Unions of balls for shape approximation in robot grasping , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[78]  Danica Kragic,et al.  Learning grasping points with shape context , 2010, Robotics Auton. Syst..

[79]  Dimitar Dimitrov,et al.  On the efficient computation of independent contact regions for force closure grasps , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[80]  Gary R. Bradski,et al.  Fast 3D recognition and pose using the Viewpoint Feature Histogram , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[81]  Pieter Abbeel,et al.  Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding , 2010, 2010 IEEE International Conference on Robotics and Automation.

[82]  Nico Blodow,et al.  General 3D modelling of novel objects from a single view , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[83]  Quoc V. Le,et al.  Learning to grasp objects with multiple contact points , 2010, 2010 IEEE International Conference on Robotics and Automation.

[84]  Kurt Konolige,et al.  Projected texture stereo , 2010, 2010 IEEE International Conference on Robotics and Automation.

[85]  Jitendra Malik,et al.  Shape matching and object recognition using shape contexts , 2010, 2010 3rd International Conference on Computer Science and Information Technology.

[86]  Jörg Stückler,et al.  Real-Time 3D Perception and Efficient Grasp Planning for Everyday Manipulation Tasks , 2011, ECMR.

[87]  Dieter Fox,et al.  A large-scale hierarchical multi-view RGB-D object dataset , 2011, 2011 IEEE International Conference on Robotics and Automation.

[88]  Dejan Pangercic,et al.  Robotic roommates making pancakes , 2011, 2011 11th IEEE-RAS International Conference on Humanoid Robots.

[89]  Danica Kragic,et al.  Multivariate discretization for Bayesian Network structure learning in robot grasping , 2011, 2011 IEEE International Conference on Robotics and Automation.

[90]  Peter Brook,et al.  Bayesian Grasp Planning , 2011 .

[91]  Markus Vincze,et al.  Pose Alignment for 3D Models and Single View Stereo Point Clouds Based on Stable Planes , 2011, 2011 International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission.

[92]  Stefan Ulbrich,et al.  The OpenGRASP benchmarking suite: An environment for the comparative analysis of grasping and dexterous manipulation , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[93]  Oliver Kroemer,et al.  Learning robot grasping from 3-D images with Markov Random Fields , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[94]  Rüdiger Dillmann,et al.  Towards shape-based visual object categorization for humanoid robots , 2011, 2011 IEEE International Conference on Robotics and Automation.

[95]  Dieter Fox,et al.  Sparse distance learning for object recognition combining RGB and depth information , 2011, 2011 IEEE International Conference on Robotics and Automation.

[96]  Matei T. Ciocarlie,et al.  Collaborative grasp planning with multiple object representations , 2011, 2011 IEEE International Conference on Robotics and Automation.

[97]  Sachin Chitta,et al.  Human-Inspired Robotic Grasp Control With Tactile Sensing , 2011, IEEE Transactions on Robotics.

[98]  Jun Morimoto,et al.  Segmentation and learning of unknown objects through physical interaction , 2011, 2011 11th IEEE-RAS International Conference on Humanoid Robots.

[99]  Nico Blodow,et al.  Combined 2D–3D categorization and classification for multimodal perception systems , 2011, Int. J. Robotics Res..

[100]  Danica Kragic,et al.  Joint Observation of Object Pose and Tactile Imprints for Online Grasp Stability Assessment , 2011, ICRA 2011.

[101]  Stefan Schaal,et al.  Learning motion primitive goals for robust manipulation , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[102]  Tamim Asfour,et al.  Planning grasps for robotic hands using a novel object representation based on the medial axis transform , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[103]  Stefan Schaal,et al.  Online movement adaptation based on previous sensor experiences , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[104]  Oussama Khatib,et al.  Grasping with application to an autonomous checkout robot , 2011, 2011 IEEE International Conference on Robotics and Automation.

[105]  Peter K. Allen,et al.  Data-driven grasping , 2011, Auton. Robots.

[106]  Niklas Bergström,et al.  Generating object hypotheses in natural scenes through human-robot interaction , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[107]  Jimmy A. Jørgensen,et al.  Grasping unknown objects using an Early Cognitive Vision system for general scene understanding , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[108]  Markus Vincze,et al.  Shape-based depth image to 3D model matching and classification with inter-view similarity , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[109]  Danica Kragic,et al.  Mind the gap - robotic grasping under incomplete observation , 2011, 2011 IEEE International Conference on Robotics and Automation.

[110]  Siddhartha S. Srinivasa,et al.  The MOPED framework: Object recognition and pose estimation for manipulation , 2011, Int. J. Robotics Res..

[111]  Danica Kragic,et al.  Task-Based Grasp Adaptation on a Humanoid Robot , 2012, SyRoCo.

[112]  Patric Jensfelt,et al.  Exploiting and modeling local 3D structure for predicting object locations , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[113]  Dmitry Berenson,et al.  Toward cloud-based grasping with uncertainty in shape: Estimating lower bounds on achieving force closure with zero-slip push grasps , 2012, 2012 IEEE International Conference on Robotics and Automation.

[114]  Vijay Kumar,et al.  Planar, bimanual, whole-arm grasping , 2012, 2012 IEEE International Conference on Robotics and Automation.

[115]  Darius Burschka,et al.  Rigid 3D geometry matching for grasping of known objects in cluttered scenes , 2012, Int. J. Robotics Res..

[116]  Daniel Leidner,et al.  Power grasp planning for anthropomorphic robot hands , 2012, 2012 IEEE International Conference on Robotics and Automation.

[117]  Danica Kragic,et al.  Visual servoing on unknown objects , 2012 .

[118]  Danica Kragic,et al.  Generalizing grasps across partly similar objects , 2012, 2012 IEEE International Conference on Robotics and Automation.

[119]  Ling Xu,et al.  Physical Human Interactive Guidance: Identifying Grasping Principles From Human-Planned Grasps , 2012, IEEE Transactions on Robotics.

[120]  Peter K. Allen,et al.  Semantic grasping: Planning robotic grasps functionally suitable for an object manipulation task , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[121]  Peter K. Allen,et al.  Pose error robust grasping from contact wrench space metrics , 2012, 2012 IEEE International Conference on Robotics and Automation.

[122]  Markus Vincze,et al.  Empty the basket - a shape based learning approach for grasping piles of unknown objects , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[123]  Danica Kragic,et al.  From object categories to grasp transfer using probabilistic reasoning , 2012, 2012 IEEE International Conference on Robotics and Automation.

[124]  Larry H. Matthies,et al.  End-to-end dexterous manipulation with deliberate interactive estimation , 2012, 2012 IEEE International Conference on Robotics and Automation.

[125]  Alberto Rodriguez,et al.  From caging to grasping , 2011, Int. J. Robotics Res..

[126]  Moritz Tenorth,et al.  The RoboEarth language: Representing and exchanging knowledge about actions, objects, and environments , 2012, 2012 IEEE International Conference on Robotics and Automation.

[127]  Oliver Kroemer,et al.  A kernel-based approach to direct action perception , 2012, 2012 IEEE International Conference on Robotics and Automation.

[128]  Anis Sahbani,et al.  An overview of 3D object grasp synthesis algorithms , 2012, Robotics Auton. Syst..

[129]  Jeffrey C. Trinkle,et al.  The application of particle filtering to grasping acquisition with visual occlusion and tactile sensing , 2012, 2012 IEEE International Conference on Robotics and Automation.

[130]  Francesc Moreno-Noguer,et al.  Using depth and appearance features for informed robot grasping of highly wrinkled clothes , 2012, 2012 IEEE International Conference on Robotics and Automation.

[131]  Jimmy A. Jørgensen,et al.  VisGraB: A benchmark for vision-based grasping , 2012, Paladyn J. Behav. Robotics.

[132]  Alexander Herzog,et al.  Template-based learning of grasp selection , 2012, 2012 IEEE International Conference on Robotics and Automation.

[133]  Antonio Bicchi,et al.  On the synthesis of feasible and prehensile robotic grasps , 2012, 2012 IEEE International Conference on Robotics and Automation.

[134]  Manuel Lopes,et al.  Active learning of visual descriptors for grasping using non-parametric smoothed beta distributions , 2012, Robotics Auton. Syst..

[135]  J. Andrew Bagnell,et al.  Robust Object Grasping using Force Compliant Motion Primitives , 2012, Robotics: Science and Systems.

[136]  Jorge Dias,et al.  Extracting data from human manipulation of objects towards improving autonomous robotic grasping , 2012, Robotics Auton. Syst..

[137]  Antonio Bicchi,et al.  On the manipulability ellipsoids of underactuated robotic hands with compliance , 2012, Robotics Auton. Syst..

[138]  Máximo A. Roa,et al.  Transferring functional grasps through contact warping and local replanning , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[139]  B. Siciliano,et al.  Visual Grasp Planning for Unknown Objects Using a Multifingered Robotic Hand , 2013, IEEE/ASME Transactions on Mechatronics.

[140]  Alvaro García Cazorla,et al.  ROS : Robot Operating System , 2013 .

[141]  Danica Kragic,et al.  Learning a dictionary of prototypical grasp-predicting parts from grasping experience , 2013, 2013 IEEE International Conference on Robotics and Automation.