Leveraging appearance priors in non-rigid registration, with application to manipulation of deformable objects

Manipulation of deformable objects is a widely applicable but challenging task in robotics. One promising nonparametric approach for this problem is trajectory transfer, in which a non-rigid registration is computed between the starting scene of the demonstration and the scene at test time. This registration is extrapolated to find a function from ℝ3 to ℝ3, which is then used to warp the demonstrated robot trajectory to generate a proposed trajectory to execute in the test scene. In prior work [1] [2], only depth information from the scenes has been used to compute this warp function. This approach ignores appearance information, but there are situations in which using both shape and appearance information is necessary for finding high quality non-rigid warp functions. In this paper, we describe an approach to learn relevant appearance information about deformable objects using deep learning, and use this additional information to improve the quality of non-rigid registration between demonstration and test scenes. Our method better registers areas of interest on deformable objects that are crucial for manipulation, such as rope crossings and towel corners and edges. We experimentally validate our approach in both simulation and in the real world, and show that the utilization of appearance information leads to a significant improvement in both selecting the best matching demonstration scene for a given test scene, and finding a high quality non-rigid registration between those two scenes.

[1]  Alexandru Patriciu,et al.  Deformation Planning for Robotic Soft Tissue Manipulation , 2009, 2009 Second International Conferences on Advances in Computer-Human Interactions.

[2]  Farida Cheriet,et al.  Landmark-Based Non-rigid Registration Via Graph Cuts , 2007, ICIAR.

[3]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[4]  Anand Rangarajan,et al.  A new point matching algorithm for non-rigid registration , 2003, Comput. Vis. Image Underst..

[5]  Vincent Lepetit,et al.  Randomized trees for real-time keypoint recognition , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[6]  Kalpana C. Jondhale,et al.  Shape matching and object recognition using shape contexts , 2010 .

[7]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[8]  Mitul Saha,et al.  Motion planning for robotic manipulation of deformable linear objects , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[9]  Antonio Torralba,et al.  SIFT Flow: Dense Correspondence across Scenes and Its Applications , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  G. Wahba.  Spline models for observational data , 1990 .

[11]  Jürgen Weese,et al.  Landmark-based elastic registration using approximating thin-plate splines , 2001, IEEE Transactions on Medical Imaging.

[12]  Pieter Abbeel,et al.  Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding , 2010, 2010 IEEE International Conference on Robotics and Automation.

[13]  Stefano Carpin,et al.  Combining imitation and reinforcement learning to fold deformable planar objects , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[14]  Wolfram Burgard,et al.  Efficient motion planning for manipulation robots in environments with deformable objects , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[15]  Dmitry Berenson,et al.  Manipulation of deformable objects without modeling and simulating deformation , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[16]  Stefan Schaal,et al.  Robot Programming by Demonstration , 2009, Springer Handbook of Robotics.

[17]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[18]  Jitendra Malik,et al.  Deformable part models are convolutional neural networks , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Henk Nijmeijer,et al.  Robot Programming by Demonstration , 2010, SIMPAR.

[20]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[21]  Pierre Payeur,et al.  Dexterous Robotic Manipulation of Deformable Objects with Multi-Sensory Feedback - a Review , 2010 .

[22]  Darwin G. Caldwell,et al.  Reinforcement Learning in Robotics: Applications and Real-World Challenges , 2013, Robotics.

[23]  Robert J. Webster,et al.  Needle Steering in 3-D Via Rapid Replanning , 2014, IEEE Transactions on Robotics.

[24]  Xiaogang Wang,et al.  Joint Deep Learning for Pedestrian Detection , 2013, 2013 IEEE International Conference on Computer Vision.

[25]  Pieter Abbeel,et al.  Learning from Demonstrations Through the Use of Non-rigid Registration , 2013, ISRR.

[26]  Vincent Lepetit,et al.  DAISY: An Efficient Dense Descriptor Applied to Wide-Baseline Stereo , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  J. Schulman,et al.  Generalization in Robotic Manipulation Through The Use of Non-Rigid Registration , 2013 .

[28]  Pieter Abbeel,et al.  Superhuman performance of surgical tasks by robots using iterative learning from human-guided demonstrations , 2010, 2010 IEEE International Conference on Robotics and Automation.

[29]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[30]  Alois Knoll,et al.  Adaptive Control for Human-Robot Skilltransfer: Trajectory Planning Based on Fluid Dynamics , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[31]  Luc Van Gool,et al.  The 2005 PASCAL Visual Object Classes Challenge , 2005, MLCW.

[32]  Nancy M. Amato,et al.  An obstacle-based rapidly-exploring random tree , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[33]  Darwin G. Caldwell,et al.  On improving the extrapolation capability of task-parameterized movement models , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[34]  Francesca Odone,et al.  Histogram intersection kernel for image classification , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[35]  Richard K. Beatson,et al.  Reconstruction and representation of 3D objects with radial basis functions , 2001, SIGGRAPH.

[36]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[37]  Jitendra Malik,et al.  Shape matching and object recognition using shape contexts , 2010, 2010 3rd International Conference on Computer Science and Information Technology.

[38]  Hinrich Schütze,et al.  Introduction to information retrieval , 2008 .

[39]  Jianguo Zhang,et al.  The PASCAL Visual Object Classes Challenge , 2006 .

[40]  S. Prima,et al.  A new efficient EM-ICP algorithm for non-linear registration of 3D point sets , 2012 .

[41]  Ankush Gupta,et al.  A case study of trajectory transfer through non-rigid registration for a simplified suturing scenario , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.