Model-Driven Feedforward Prediction for Manipulation of Deformable Objects

Robotic manipulation of deformable objects is a difficult problem especially because of the complexity of the many different ways an object can deform. Searching such a high-dimensional state space makes it difficult to recognize, track, and manipulate deformable objects. In this paper, we introduce a predictive, model-driven approach to address this challenge, using a precomputed, simulated database of deformable object models. Mesh models of common deformable garments are simulated with the garments picked up in multiple different poses under gravity, and stored in a database for fast and efficient retrieval. To validate this approach, we developed a comprehensive pipeline for manipulating clothing as in a typical laundry task. First, the database is used for category and the pose estimation is used for a garment in an arbitrary position. A fully featured 3-D model of the garment is constructed in real time, and volumetric features are then used to obtain the most similar model in the database to predict the object category and pose. Second, the database can significantly benefit the manipulation of deformable objects via nonrigid registration, providing accurate correspondences between the reconstructed object model and the database models. Third, the accurate model simulation can also be used to optimize the trajectories for the manipulation of deformable objects, such as the folding of garments. Extensive experimental results are shown for the above tasks using a variety of different clothings. Note to Practitioners—This paper provides an open source, extensible, 3-D database for dissemination to the robotics and graphics communities. Model-driven methods are proliferating, and they need to be applied, tested, and validated in real environments. A key idea we have exploited is to have an innovative and novel use of simulation. This database will serve as infrastructure for developing advanced robotic machine learning algorithms. We want to address this machine learning idea ourselves, but we expect the dissemination of the database to other researchers with different agendas and task applications, which will bring wide progress in this area. Our proposed methods, as mentioned earlier, can be easily applied to interrelated areas. One example is that the 3-D shape-based matching algorithm can be used for other objects, such as bottles, papers, and food. After integrating with other robotic systems, the use of the robot can be easily extended to other tasks, such as making food, cleaning room, and fetching objects, to assist our daily life.

[1]  Nobuyuki Kita,et al.  Clothes handling based on recognition by strategic observation , 2011, 2011 11th IEEE-RAS International Conference on Humanoid Robots.

[2]  Václav Hlavác,et al.  Polygonal Models for Clothing , 2014, TAROS.

[3]  Ian D. Walker,et al.  A new approach to clothing classification using mid-level layers , 2013, 2013 IEEE International Conference on Robotics and Automation.

[4]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[5]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[6]  Ben Taskar,et al.  Object detection via boundary structure segmentation , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[7]  Thorsten Joachims,et al.  Optimizing search engines using clickthrough data , 2002, KDD.

[8]  James F. O'Brien,et al.  Bringing clothing into desired configurations with limited perception , 2011, 2011 IEEE International Conference on Robotics and Automation.

[9]  Danfei Xu,et al.  Multi-sensor surface analysis for robotic ironing , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[10]  Tae-Kyun Kim,et al.  Autonomous active recognition and unfolding of clothes using random decision forests and probabilistic planning , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[11]  Gary K. L. Tam,et al.  Registration of 3D Point Clouds and Meshes: A Survey from Rigid to Nonrigid , 2013, IEEE Transactions on Visualization and Computer Graphics.

[12]  Ulrich Eckhardt,et al.  Shape descriptors for non-rigid shapes with a single closed contour , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[13]  Carme Torras,et al.  Robot-Aided Cloth Classification Using Depth Information and CNNs , 2016, AMDO.

[14]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[15]  Trevor Darrell,et al.  A geometric approach to robotic laundry folding , 2012, Int. J. Robotics Res..

[16]  Pieter Abbeel,et al.  Tracking deformable objects with point clouds , 2013, 2013 IEEE International Conference on Robotics and Automation.

[17]  Björn Stenger,et al.  Shape context and chamfer matching in cluttered scenes , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[18]  Daniel P. Huttenlocher,et al.  Comparing Images Using the Hausdorff Distance , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  Zhuowen Tu,et al.  Shape Matching and Recognition - Using Generative Models and Informative Features , 2004, ECCV.

[20]  Shih-Fu Chang,et al.  Real-time pose estimation of deformable objects using a volumetric approach , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[21]  Ioannis Mariolis,et al.  Pose and category recognition of highly deformable objects using deep learning , 2015, 2015 International Conference on Advanced Robotics (ICAR).

[22]  Hao Li,et al.  Global Correspondence Optimization for Non‐Rigid Registration of Depth Scans , 2008, Comput. Graph. Forum.

[23]  Peter K. Allen,et al.  Recognition of deformable object category and pose , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[24]  Kaj Madsen,et al.  Methods for Non-Linear Least Squares Problems , 1999 .

[25]  Matteo Zoppi,et al.  On the Development of a Specialized Flexible Gripper for Garment Handling , 2013 .

[26]  Mathieu Desbrun,et al.  Discrete shells , 2003, SCA '03.

[27]  Danfei Xu,et al.  Folding deformable objects using predictive simulation and trajectory optimization , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[28]  Bernard Chazelle,et al.  Matching 3D models with shape distributions , 2001, Proceedings International Conference on Shape Modeling and Applications.

[29]  Kaj Madsen,et al.  Methods for Non-Linear Least Squares Problems (2nd ed.) , 2004 .

[30]  Shih-Fu Chang,et al.  Regrasping and unfolding of garments using predictive thin shell modeling , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[31]  Jitendra Malik,et al.  Shape matching and object recognition using shape contexts , 2010, 2010 3rd International Conference on Computer Science and Information Technology.

[32]  Ian D. Walker,et al.  Classification of clothing using interactive perception , 2011, 2011 IEEE International Conference on Robotics and Automation.

[33]  Vladimír Petrík,et al.  Garment perception and its folding using a dual-arm robot , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[34]  Nobuyuki Kita,et al.  Clothes state recognition using 3D observed data , 2009, 2009 IEEE International Conference on Robotics and Automation.

[35]  Hiroaki Seki,et al.  Unfolding of Massive Laundry and Classification Types by Dual Manipulator , 2007, J. Adv. Comput. Intell. Intell. Informatics.

[36]  Trevor Darrell,et al.  Parametrized shape models for clothing , 2011, 2011 IEEE International Conference on Robotics and Automation.

[37]  Y. Tsai Rapid and accurate computation of the distance function using grids , 2002 .

[38]  Matei T. Ciocarlie,et al.  The Columbia grasp database , 2009, 2009 IEEE International Conference on Robotics and Automation.

[39]  Rongrong Ji,et al.  Label Propagation from ImageNet to 3D Point Clouds , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[40]  Jiawen Chen,et al.  Scalable real-time volumetric surface reconstruction , 2013, ACM Trans. Graph..

[41]  Nobuyuki Kita,et al.  A model-driven method of estimating the state of clothes for manipulating it , 2002, Sixth IEEE Workshop on Applications of Computer Vision, 2002. (WACV 2002). Proceedings..

[42]  Jun Wang,et al.  3D Facial Expression Recognition Based on Primitive Surface Feature Distribution , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[43]  Jonathan T. Barron,et al.  3D self-portraits , 2013, ACM Trans. Graph..

[44]  Li Sun,et al.  Accurate garment surface analysis using an active stereo robot head with application to dual-arm flattening , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[45]  Pieter Abbeel,et al.  Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding , 2010, 2010 IEEE International Conference on Robotics and Automation.

[46]  Trevor Darrell,et al.  Perception for the manipulation of socks , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[47]  Pieter Abbeel,et al.  Gravity-Based Robotic Cloth Folding , 2010, WAFR.

[48]  Jan-Michael Frahm,et al.  3D model matching with Viewpoint-Invariant Patches (VIP) , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[49]  Jitendra Malik,et al.  Recognizing Objects in Range Data Using Regional Point Descriptors , 2004, ECCV.

[50]  Fujio Yamaguchi,et al.  Curves and Surfaces in Computer Aided Geometric Design , 1988, Springer Berlin Heidelberg.