Primitive Shape Recognition for Object Grasping

Shape informs how an object should be grasped, both in terms of where and how. As such, this paper describes a segmentation-based architecture for decomposing objects sensed with a depth camera into multiple primitive shapes, along with a post-processing pipeline for robotic grasping. Segmentation employs a deep network, called PS-CNN, trained on synthetic data with 6 classes of primitive shapes and generated using a simulation engine. Each primitive shape is designed with parametrized grasp families, permitting the pipeline to identify multiple grasp candidates per shape region. The grasps are rank ordered, with the first feasible one chosen for execution. For task-free grasping of individual objects, the method achieves a 94.2% success rate placing it amongst the top performing grasp methods when compared to top-down and SE(3)-based approaches. Additional tests involving variable viewpoints and clutter demonstrate robustness to setup. For task-oriented grasping, PS-CNN achieves a 93.0% success rate. Overall, the outcomes support the hypothesis that explicitly encoding shape primitives within a grasping pipeline should boost grasping performance, including task-free and task-relevant grasp prediction.

[1]  Subhransu Maji,et al.  ParSeNet: A Parametric Surface Fitting Network for 3D Point Clouds , 2020, ECCV.

[2]  Francesc Moreno-Noguer,et al.  Multi-FinGAN: Generative Coarse-To-Fine Sampling of Multi-Finger Grasps , 2020, IEEE International Conference on Robotics and Automation.

[3]  Henrik I. Christensen,et al.  Automatic grasp planning using shape primitives , 2003, 2003 IEEE International Conference on Robotics and Automation (Cat. No.03CH37422).

[4]  Lihi Zelnik-Manor,et al.  How to Evaluate Foreground Maps , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Helge Ritter,et al.  MoveIt! Task Constructor for Task-Level Motion Planning , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[6]  Silvio Savarese,et al.  Learning task-oriented grasping for tool manipulation from simulated self-supervision , 2018, Robotics: Science and Systems.

[7]  Stefan Leutenegger,et al.  Deep learning a grasp function for grasping under gripper pose uncertainty , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[8]  Slobodan Ilic,et al.  DeceptionNet: Network-Driven Domain Randomization , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[9]  Mohammed Bennamoun,et al.  RGB-D Object Recognition and Grasp Detection Using Hierarchical Cascaded Forests , 2017, IEEE Transactions on Robotics.

[10]  Kate Saenko,et al.  Grasp Pose Detection in Point Clouds , 2017, Int. J. Robotics Res..

[11]  Honglak Lee,et al.  Deep learning for detecting robotic grasps , 2013, Int. J. Robotics Res..

[12]  D. Marr,et al.  Representation and recognition of the spatial organization of three-dimensional shapes , 1978, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[13]  Vibhav Vineet,et al.  Photorealistic Image Synthesis for Object Instance Detection , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[14]  Patricio A. Vela,et al.  Using Synthetic Data and Deep Networks to Recognize Primitive Shapes for Object Grasping , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[15]  Sergey Levine,et al.  Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection , 2016, Int. J. Robotics Res..

[16]  Dieter Fox,et al.  PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes , 2017, Robotics: Science and Systems.

[17]  Andreas Geiger,et al.  Learning Unsupervised Hierarchical Part Decomposition of 3D Objects From a Single RGB Image , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Leonidas J. Guibas,et al.  Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Hideo Saito,et al.  Pose estimation of primitive-shaped objects from a depth image using superquadric representation , 2020 .

[20]  Stephen Tyree,et al.  NViSII: A Scriptable Tool for Photorealistic Image Generation , 2021, ArXiv.

[21]  Ken Goldberg,et al.  Segmenting Unknown 3D Objects from Real Depth Images using Mask R-CNN Trained on Synthetic Data , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[22]  Kenneth Y. Goldberg,et al.  Learning Deep Policies for Robot Bin Picking by Simulating Robust Grasping Sequences , 2017, CoRL.

[23]  Hao Su,et al.  S4G: Amodal Single-view Single-Shot SE(3) Grasp Detection in Cluttered Scenes , 2019, CoRL.

[24]  Danica Kragic,et al.  Selection of robot pre-grasps using box-based shape approximation , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[25]  Leonidas J. Guibas,et al.  Supervised Fitting of Geometric Primitives to 3D Point Clouds , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Swami Sankaranarayanan,et al.  Learning from Synthetic Data: Addressing Domain Shift for Semantic Segmentation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[27]  Avinash C. Kak,et al.  Fast construction of force-closure grasps , 1996, IEEE Trans. Robotics Autom..

[28]  Subhransu Maji,et al.  3D Shape Segmentation with Projective Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Andreas Geiger,et al.  Superquadrics Revisited: Learning 3D Shape Parsing Beyond Cuboids , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Ming Ouhyoung,et al.  On Visual Similarity Based 3D Model Retrieval , 2003, Comput. Graph. Forum.

[31]  Danica Kragic,et al.  Data-Driven Grasp Synthesis—A Survey , 2013, IEEE Transactions on Robotics.

[32]  Jianbin Tang,et al.  Densely Supervised Grasp Detector (DSGD) , 2018, AAAI.

[33]  Oliver Brock,et al.  Grasping unknown objects by exploiting shape adaptability and environmental constraints , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[34]  Patricio A. Vela,et al.  GKNet: Grasp keypoint network for grasp candidates detection , 2021, Int. J. Robotics Res..

[35]  Wojciech Zaremba,et al.  Domain Randomization and Generative Models for Robotic Grasping , 2017, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[36]  Antonio Bicchi,et al.  On the synthesis of feasible and prehensile robotic grasps , 2012, 2012 IEEE International Conference on Robotics and Automation.

[37]  Dieter Fox,et al.  6-DOF GraspNet: Variational Grasp Generation for Object Manipulation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[38]  Reinhard Klein,et al.  Efficient RANSAC for Point‐Cloud Shape Detection , 2007, Comput. Graph. Forum.

[39]  Peter Corke,et al.  Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach , 2018, Robotics: Science and Systems.

[40]  David P. Dobkin,et al.  A search engine for 3D models , 2003, TOGS.

[41]  Ken Goldberg,et al.  Learning ambidextrous robot grasping policies , 2019, Science Robotics.

[42]  Vijay Kumar,et al.  Robotic grasping and contact: a review , 2000, Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065).

[43]  Leonidas J. Guibas,et al.  Learning Shape Abstractions by Assembling Volumetric Primitives , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Xinyu Liu,et al.  Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics , 2017, Robotics: Science and Systems.

[45]  Markus Vincze,et al.  3DNet: Large-scale object class recognition from CAD models , 2012, 2012 IEEE International Conference on Robotics and Automation.

[46]  Jian Chen,et al.  Grasp Proposal Networks: An End-to-End Solution for Visual Learning of Robotic Grasps , 2020, NeurIPS.

[47]  Alberto Rodriguez,et al.  Learning Synergies Between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[48]  Yunzhou Zhang,et al.  Reasonable Grasping Based on Hierarchical Decomposition Models of Unknown Objects , 2018, 2018 15th International Conference on Control, Automation, Robotics and Vision (ICARCV).

[49]  Peter I. Corke,et al.  The ACRV picking benchmark: A robotic shelf picking benchmark to foster reproducible research , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[50]  Ashutosh Saxena,et al.  Efficient grasping from RGBD images: Learning using a new rectangle representation , 2011, 2011 IEEE International Conference on Robotics and Automation.

[51]  Qixin Cao,et al.  A New Approach Based on Two-stream CNNs for Novel Objects Grasping in Clutter , 2019, J. Intell. Robotic Syst..

[52]  Thomas A. Funkhouser,et al.  The Princeton Shape Benchmark , 2004, Proceedings Shape Modeling Applications, 2004..

[53]  Yangtao Zheng,et al.  Double-Dot Network for Antipodal Grasp Detection , 2021, 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[54]  Wojciech Zaremba,et al.  Domain randomization for transferring deep neural networks from simulation to the real world , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[55]  Dongwon Park,et al.  Classification based Grasp Detection using Spatial Transformer Network , 2018, ArXiv.

[56]  Kate Saenko,et al.  Learning a visuomotor controller for real world robotic grasping using simulated depth images , 2017, CoRL.

[57]  Robert Platt,et al.  Using Geometry to Detect Grasp Poses in 3D Point Clouds , 2015, ISRR.

[58]  Emmanuel Dellandréa,et al.  Jacquard: A Large Scale Dataset for Robotic Grasp Detection , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[59]  Marco Attene,et al.  Hierarchical Structure Recovery of Point‐Sampled Surfaces , 2010, Comput. Graph. Forum.

[60]  Brenna Argall,et al.  Grasp detection for assistive robotic manipulation , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[61]  Hujun Bao,et al.  PVNet: Pixel-Wise Voting Network for 6DoF Pose Estimation , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[62]  Dieter Fox,et al.  Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects , 2018, CoRL.

[63]  Lorenzo Natale,et al.  A grasping approach based on superquadric models , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[64]  Patricio A. Vela,et al.  Real-World Multiobject, Multigrasp Detection , 2018, IEEE Robotics and Automation Letters.

[65]  Peter K. Allen,et al.  Grasp Planning via Decomposition Trees , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[66]  Fuchun Sun,et al.  PointNetGPD: Detecting Grasp Configurations from Point Sets , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[67]  Ken Goldberg,et al.  On-Policy Dataset Synthesis for Learning Robot Grasping Policies Using Fully Convolutional Deep Networks , 2019, IEEE Robotics and Automation Letters.

[68]  Natsuki Yamanobe,et al.  Grasp planning for everyday objects based on primitive shape representation for parallel jaw grippers , 2010, 2010 IEEE International Conference on Robotics and Biomimetics.

[69]  Sven Behnke,et al.  Real-Time Plane Segmentation Using RGB-D Cameras , 2012, RoboCup.

[70]  Stefano Caselli,et al.  A 3D shape segmentation approach for robot grasping by parts , 2012, Robotics Auton. Syst..

[71]  Ales Leonardis,et al.  One-shot learning and generation of dexterous grasps for novel objects , 2016, Int. J. Robotics Res..

[72]  Stanley T. Birchfield,et al.  Falling Things: A Synthetic Dataset for 3D Object Detection and Pose Estimation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[73]  Douglas Chai,et al.  Review of Deep Learning Methods in Robotic Grasp Detection , 2018, Multimodal Technol. Interact..

[74]  Stanley T. Birchfield,et al.  Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[75]  Amelia Carolina Sparavigna,et al.  CLD-shaped Brushstrokes in Non-Photorealistic Rendering , 2010, ArXiv.

[76]  Marcin Andrychowicz,et al.  Sim-to-Real Transfer of Robotic Control with Dynamics Randomization , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[77]  Stefanie Tellex,et al.  Learning to Detect Multi-Modal Grasps for Dexterous Grasping in Dense Clutter , 2021, 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[78]  Surya P. N. Singh,et al.  V-REP: A versatile and scalable robot simulation framework , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[79]  Russ Tedrake,et al.  A Supervised Approach to Predicting Noise in Depth Images , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[80]  Wenguang Zhang,et al.  PointNet++ Grasping: Learning An End-to-end Spatial Grasp Generation Algorithm from Sparse Point Clouds , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[81]  Fumiya Iida,et al.  Real-World, Real-Time Robotic Grasping with Convolutional Neural Networks , 2017, TAROS.

[82]  Siddhartha Mukherjee Study on performance improvement of oil paint image filter algorithm using parallel pattern library , 2014, ArXiv.

[83]  Tucker Hermans,et al.  Planning Multi-Fingered Grasps as Probabilistic Inference in a Learned Deep Network , 2018, ISRR.

[84]  Yang Yang,et al.  Learning to Generate 6-DoF Grasp Poses with Reachability Awareness , 2019, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[85]  P. Abbeel,et al.  Yale-CMU-Berkeley dataset for robotic manipulation research , 2017, Int. J. Robotics Res..

[86]  Akira Nakamura,et al.  Modeling of everyday objects for semantic grasp , 2014, The 23rd IEEE International Symposium on Robot and Human Interactive Communication.

[87]  Abhinav Gupta,et al.  Supersizing self-supervision: Learning to grasp from 50K tries and 700 robot hours , 2015, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[88]  Richard M. Murray,et al.  A Mathematical Introduction to Robotic Manipulation , 1994 .

[89]  Kate Saenko,et al.  High precision grasp pose detection in dense clutter , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[90]  Jakub W. Pachocki,et al.  Learning dexterous in-hand manipulation , 2018, Int. J. Robotics Res..

[91]  Tamy Boubekeur,et al.  A Survey of Simple Geometric Primitives Detection Methods for Captured 3D Data , 2018, Comput. Graph. Forum.

[92]  Antonio Bicchi,et al.  On the manipulability ellipsoids of underactuated robotic hands with compliance , 2012, Robotics Auton. Syst..