Learning to Smooth and Fold Real Fabric Using Dense Object Descriptors Trained on Synthetic Color Images

Robotic fabric manipulation is challenging due to the infinite dimensional configuration space and complex dynamics. In this paper, we learn visual representations of deformable fabric by training dense object descriptors that capture correspondences across images of fabric in various configurations. The learned descriptors capture higher level geometric structure, facilitating design of explainable policies. We demonstrate that the learned representation facilitates multistep fabric smoothing and folding tasks on two real physical systems, the da Vinci surgical robot and the ABB YuMi given high level demonstrations from a supervisor. The system achieves a 78.8% average task success rate across six fabric manipulation tasks. See this https URL for supplementary material and videos.

[1]  Russ Tedrake,et al.  Self-Supervised Correspondence in Visuomotor Policy Learning , 2019, IEEE Robotics and Automation Letters.

[2]  Dinesh Manocha,et al.  Manipulating Highly Deformable Materials Using a Visual Feedback Dictionary , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[3]  Russ Tedrake,et al.  Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation , 2018, CoRL.

[4]  Nobuyuki Kita,et al.  Clothes state recognition using 3D observed data , 2009, 2009 IEEE International Conference on Robotics and Automation.

[5]  Pieter Abbeel,et al.  Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding , 2010, 2010 IEEE International Conference on Robotics and Automation.

[6]  Geoffrey J. Gordon,et al.  A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning , 2010, AISTATS.

[7]  Hiroaki Seki,et al.  Unfolding of Massive Laundry and Classification Types by Dual Manipulator , 2007, J. Adv. Comput. Intell. Intell. Informatics.

[8]  Carme Torras,et al.  A Grasping-Centered Analysis for Cloth Manipulation , 2019, IEEE Transactions on Robotics.

[9]  John Canny,et al.  Deep Imitation Learning of Sequential Fabric Smoothing From an Algorithmic Supervisor , 2019 .

[10]  Trevor Darrell,et al.  A geometric approach to robotic laundry folding , 2012, Int. J. Robotics Res..

[11]  Soshi Iba,et al.  Deep Transfer Learning of Pick Points on Fabric for Robot Bed-Making , 2018, ISRR.

[12]  Xavier Provot,et al.  Deformation Constraints in a Mass-Spring Model to Describe Rigid Cloth Behavior , 1995 .

[13]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[14]  Priya Sundaresan,et al.  Learning Rope Manipulation Policies Using Dense Object Descriptors Trained on Synthetic Depth Data , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[15]  Jacob Rosen,et al.  Autonomous Tissue Manipulation via Surgical Robot Using Learning Based Model Predictive Control , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[16]  Oliver Kroemer,et al.  A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms , 2019, J. Mach. Learn. Res..

[17]  Sergey Levine,et al.  QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation , 2018, CoRL.

[18]  Brijen Thananjeyan,et al.  Safety Augmented Value Estimation From Demonstrations (SAVED): Safe Deep Model-Based RL for Sparse Cost Robotic Tasks , 2020, IEEE Robotics and Automation Letters.

[19]  Ken Goldberg,et al.  Motion2Vec: Semi-Supervised Representation Learning from Surgical Videos , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[20]  Peter Kazanzides,et al.  An open-source research kit for the da Vinci® Surgical System , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[21]  Li Sun,et al.  Accurate garment surface analysis using an active stereo robot head with application to dual-arm flattening , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[22]  Wojciech Zaremba,et al.  Domain randomization for transferring deep neural networks from simulation to the real world , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[23]  Dieter Fox,et al.  Self-Supervised Visual Descriptor Learning for Dense Correspondence , 2017, IEEE Robotics and Automation Letters.

[24]  Stefano Carpin,et al.  Combining imitation and reinforcement learning to fold deformable planar objects , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[25]  Sergey Levine,et al.  (CAD)$^2$RL: Real Single-Image Flight without a Single Real Image , 2016, Robotics: Science and Systems.

[26]  Sergey Levine,et al.  Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection , 2016, Int. J. Robotics Res..

[27]  Xavier Provot,et al.  Collision and self-collision handling in cloth model dedicated to design garments , 1997, Computer Animation and Simulation.

[28]  J. Paul Siebert,et al.  A Heuristic-Based Approach for Flattening Wrinkled Clothes , 2013, TAROS.

[29]  Ian D. Walker,et al.  Model for unfolding laundry using interactive perception , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[30]  Nobuyuki Kita,et al.  A method for handling a specific part of clothing by dual arms , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[31]  Tae-Kyun Kim,et al.  Autonomous active recognition and unfolding of clothes using random decision forests and probabilistic planning , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[32]  Pieter Abbeel,et al.  Learning to Manipulate Deformable Objects without Demonstrations , 2019, Robotics: Science and Systems.

[33]  Carme Torras,et al.  Dynamic Cloth Manipulation with Deep Reinforcement Learning , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[34]  Brijen Thananjeyan,et al.  Multilateral surgical pattern cutting in 2D orthotropic gauze with deep reinforcement learning policies for tensioning , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[35]  Abhinav Gupta,et al.  Supersizing self-supervision: Learning to grasp from 50K tries and 700 robot hours , 2015, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[36]  Dinesh Manocha,et al.  Cloth Manipulation Using Random-Forest-Based Imitation Learning , 2018, IEEE Robotics and Automation Letters.

[37]  Sergey Levine,et al.  Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor , 2018, ICML.

[38]  Shuran Song,et al.  Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[39]  Andrew J. Davison,et al.  Sim-to-Real Reinforcement Learning for Deformable Object Manipulation , 2018, CoRL.

[40]  Katsu Yamane,et al.  VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation , 2020, RSS 2020.

[41]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[42]  Frank W. Paul,et al.  Vision-guided robotic fabric manipulation for apparel manufacturing , 1987, IEEE Control Systems Magazine.

[43]  Rajiv Dubey,et al.  Robotic Fabric Handling for Automating Garment Manufacturing , 1983 .

[44]  Pieter Abbeel,et al.  Learning from Demonstrations Through the Use of Non-rigid Registration , 2013, ISRR.

[45]  Ken Goldberg,et al.  Learning ambidextrous robot grasping policies , 2019, Science Robotics.

[46]  Belhassen-Chedli Bouzgarrou,et al.  Robotic manipulation and sensing of deformable objects in domestic and industrial applications: a survey , 2018, Int. J. Robotics Res..

[47]  Pieter Abbeel,et al.  Learning Predictive Representations for Deformable Objects Using Contrastive Estimation , 2020, CoRL.

[48]  Sergey Levine,et al.  Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control , 2018, ArXiv.