Modeling of Deformable Objects for Robotic Manipulation: A Tutorial and Review

Manipulation of deformable objects has given rise to an important set of open problems in the field of robotics. Application areas include robotic surgery, household robotics, manufacturing, logistics, and agriculture, to name a few. Related research problems span modeling and estimation of an object's shape, estimation of an object's material properties, such as elasticity and plasticity, object tracking and state estimation during manipulation, and manipulation planning and control. In this survey article, we start by providing a tutorial on foundational aspects of models of shape and shape dynamics. We then use this as the basis for a review of existing work on learning and estimation of these models and on motion planning and control to achieve desired deformations. We also discuss potential future lines of work.

[1]  C. D. Boor,et al.  Splines as linear combinations of B-splines. A Survey , 1976 .

[2]  A. Kacelnik,et al.  A New Caledonian crow (Corvus moneduloides) creatively re-designs tools by bending or unbending aluminium strips , 2006, Animal Cognition.

[3]  Lydia E. Kavraki,et al.  Path planning for minimal energy curves of constant length , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[4]  Danica Kragic,et al.  Data-Driven Grasp Synthesis—A Survey , 2013, IEEE Transactions on Robotics.

[5]  Thomas W. Sederberg,et al.  Free-form deformation of solid geometric models , 1986, SIGGRAPH.

[6]  Kazuaki Iwata,et al.  Static analysis of deformable object grasping based on bounded force closure , 1996, Proceedings of IEEE International Conference on Robotics and Automation.

[7]  Fernando Torres Medina,et al.  Tactile control based on Gaussian images and its application in bi-manual manipulation of deformable objects , 2017, Robotics Auton. Syst..

[8]  Xin Tong,et al.  A scalable galerkin multigrid method for real-time simulation of deformable objects , 2019, ACM Trans. Graph..

[9]  M. Romeo,et al.  Muscle and Fascia Simulation with Extended Position Based Dynamics , 2020, Comput. Graph. Forum.

[10]  Alex Pentland,et al.  Good vibrations: modal dynamics for graphics and animation , 1989, SIGGRAPH.

[11]  Yunhui Liu,et al.  Uncalibrated vision-based deformation control of compliant objects with online estimation of the Jacobian matrix , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[12]  Yunhui Liu,et al.  On the visual deformation servoing of compliant objects: Uncalibrated control methods and experiments , 2014, Int. J. Robotics Res..

[13]  W. Eric L. Grimson,et al.  Model-based curve evolution technique for image segmentation , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[14]  Olivier D. Faugeras,et al.  Statistical shape influence in geodesic active contours , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[15]  Scott Schaefer,et al.  Example-based skeleton extraction , 2007, Symposium on Geometry Processing.

[16]  Daniel L. K. Yamins,et al.  Flexible Neural Representation for Physics Prediction , 2018, NeurIPS.

[17]  Vitaly Kober,et al.  An efficient algorithm for non-rigid object registration , 2020 .

[18]  Belhassen-Chedli Bouzgarrou,et al.  Model-based strategy for grasping 3D deformable objects using a multi-fingered robotic hand , 2017, Robotics Auton. Syst..

[19]  Li Bai,et al.  3D Modeling for Deformable Objects , 2008, AMDO.

[20]  Dinesh Manocha,et al.  3-D Deformable Object Manipulation Using Deep Neural Networks , 2019, IEEE Robotics and Automation Letters.

[21]  Fernando Torres Medina,et al.  Study of dexterous robotic grasping for deformable objects manipulation , 2015, 2015 23rd Mediterranean Conference on Control and Automation (MED).

[22]  Kaspar Althoefer,et al.  Stable Grip Control on Soft Objects With Time-Varying Stiffness , 2016, IEEE Transactions on Robotics.

[23]  Mitul Saha,et al.  Manipulation Planning for Deformable Linear Objects , 2007, IEEE Transactions on Robotics.

[24]  Ronald Fedkiw,et al.  Finite volume methods for the simulation of skeletal muscle , 2003, SCA '03.

[25]  Jackie Chappell,et al.  Salient Features and Snapshots in Time: An Interdisciplinary Perspective on Object Representation , 2013 .

[26]  Danica Kragic,et al.  Estimating the deformability of elastic materials using optical flow and position-based dynamics , 2015, 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids).

[27]  Haisheng Li,et al.  Fast template matching based on deformable best-buddies similarity measure , 2018, Multimedia Tools and Applications.

[28]  Pieter Abbeel,et al.  Tracking deformable objects with point clouds , 2013, 2013 IEEE International Conference on Robotics and Automation.

[29]  Bo Zhu,et al.  Simulation of organ deformation using boundary element method and meshless shape matching , 2008, 2008 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[30]  Demetri Terzopoulos,et al.  Snakes: Active contour models , 2004, International Journal of Computer Vision.

[31]  Ana-Maria Cretu,et al.  Estimation of deformable object properties from shape and force measurements for virtualized reality applications , 2010, 2010 IEEE International Symposium on Haptic Audio Visual Environments and Games.

[32]  Wolfram Burgard,et al.  Learning object deformation models for robot motion planning , 2014, Robotics Auton. Syst..

[33]  Guido Gerig,et al.  Segmentation of 3D Objects from MRI Volume Data Using Constrained Elastic Deformations of Flexible Fourier Surface Models , 1995, CVRMed.

[34]  Gábor Székely,et al.  Simultaneous Topology and Stiffness Identification for Mass-Spring Models Based on FEM Reference Deformations , 2004, MICCAI.

[35]  Anurag Mittal,et al.  Multi-stage Contour Based Detection of Deformable Objects , 2008, ECCV.

[36]  Shih-Fu Chang,et al.  Model-Driven Feedforward Prediction for Manipulation of Deformable Objects , 2016, IEEE Transactions on Automation Science and Engineering.

[37]  Alex Pentland,et al.  A Unified Approach for Physical and Geometric Modeling for Graphics and Animation , 1992, Comput. Graph. Forum.

[38]  P. Moore,et al.  A Survey of Computer-Based Deformable Models , 2007, International Machine Vision and Image Processing Conference (IMVIP 2007).

[39]  Alan S. Willsky,et al.  Learning the Dynamics and Time-Recursive Boundary Detection of Deformable Objects , 2008, IEEE Transactions on Image Processing.

[40]  Andrew W. Fitzgibbon,et al.  Finding nemo: Deformable object class modelling using curve matching , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[41]  Paolo Fiorini,et al.  Trajectory planning with task constraints in densely filled environments , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[42]  Danica Kragic,et al.  Active exploration using Gaussian Random Fields and Gaussian Process Implicit Surfaces , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[43]  Shinichi Hirai,et al.  Robust manipulation of deformable objects by a simple PID feedback , 2001, Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164).

[44]  Eigil Samset,et al.  Bayesian Estimation of Deformation and Elastic Parameters in Non-rigid Registration , 2010, WBIR.

[45]  Jadav Das,et al.  Autonomous Shape Control of a Deformable Object by Multiple Manipulators , 2011, J. Intell. Robotic Syst..

[46]  Stephane Cotin,et al.  Capturing Deformations of Interacting Non-rigid Objects Using RGB-D Data , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[47]  Michael Unser,et al.  Splines: a perfect fit for signal and image processing , 1999, IEEE Signal Process. Mag..

[48]  Razvan Pascanu,et al.  Interaction Networks for Learning about Objects, Relations and Physics , 2016, NIPS.

[49]  Andrew J. Davison,et al.  Live dense reconstruction with a single moving camera , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[50]  I. Demirdzic,et al.  Thirty Years of the Finite Volume Method for Solid Mechanics , 2018, Archives of Computational Methods in Engineering.

[51]  Song Han,et al.  On Recovering Hyperquadrics from Range Data , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[52]  Yunhui Liu,et al.  Automatic 3-D Manipulation of Soft Objects by Robotic Arms With an Adaptive Deformation Model , 2016, IEEE Transactions on Robotics.

[53]  Lydia E. Kavraki,et al.  Deformable volumes in path planning applications , 2000, Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065).

[54]  Brian Mirtich,et al.  A Survey of Deformable Modeling in Computer Graphics , 1997 .

[55]  Dinesh Manocha,et al.  Active Animations of Reduced Deformable Models with Environment Interactions , 2017, ACM Trans. Graph..

[56]  Alexandru Patriciu,et al.  Deformation Planning for Robotic Soft Tissue Manipulation , 2009, 2009 Second International Conferences on Advances in Computer-Human Interactions.

[57]  Theodore Kim,et al.  Optimizing cubature for efficient integration of subspace deformations , 2008, SIGGRAPH Asia '08.

[58]  Zhihua Wang,et al.  3D-PhysNet: Learning the Intuitive Physics of Non-Rigid Object Deformations , 2018, IJCAI.

[59]  Johan Montagnat,et al.  A review of deformable surfaces: topology, geometry and deformation , 2001, Image Vis. Comput..

[60]  Markus H. Gross,et al.  Meshless deformations based on shape matching , 2005, ACM Trans. Graph..

[61]  Bryan M. Williams,et al.  Learning Active Contour Models for Medical Image Segmentation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[62]  Jeremy L. Wyatt,et al.  A Multimodal Model of Object Deformation Under Robotic Pushing , 2017, IEEE Transactions on Cognitive and Developmental Systems.

[63]  Adrien Bartoli,et al.  Shape-From-Template with Curves , 2019, International Journal of Computer Vision.

[64]  Yan-Bin Jia,et al.  Picking up a soft 3D object by “feeling” the grip , 2015, Int. J. Robotics Res..

[65]  Masaru Ishii,et al.  The deformable most-likely-point paradigm , 2019, Medical Image Anal..

[66]  Fernando Torres,et al.  Adaptive tactile control for in-hand manipulation tasks of deformable objects , 2017 .

[67]  Lang Xu,et al.  Integrating viscoelastic mass spring dampers into position-based dynamics to simulate soft tissue deformation in real time , 2018, Royal Society Open Science.

[68]  Ana-Maria Cretu,et al.  Soft Object Deformation Monitoring and Learning for Model-Based Robotic Hand Manipulation , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[69]  Ron Alterovitz,et al.  Motion Planning Under Uncertainty In Highly Deformable Environments , 2011, Robotics: Science and Systems.

[70]  Mahdi A. Nisirat,et al.  A New External Force for Snake Algorithm Based on Energy Diffusion , 2019, International Journal of Machine Learning and Computing.

[71]  Juan Manuel Cordero Valle,et al.  Curvas y superficies para modelado geométrico , 2002 .

[72]  Bradley J. Nelson,et al.  A Deformable Object Tracking Algorithm Based on the Boundary Element Method that is Robust to Occlusions and Spurious Edges , 2008, International Journal of Computer Vision.

[73]  Danica Kragic,et al.  Trends and challenges in robot manipulation , 2019, Science.

[74]  Jérémie Dequidt,et al.  Real-time control of soft-robots using asynchronous finite element modeling , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[75]  Andrew Nealen,et al.  Physically Based Deformable Models in Computer Graphics , 2006, Comput. Graph. Forum.

[76]  Timothy F. Cootes,et al.  Statistical models of appearance for computer vision , 1999 .

[77]  E. Catmull,et al.  Recursively generated B-spline surfaces on arbitrary topological meshes , 1978 .

[78]  Yongmin Zhong,et al.  Neural dynamics-based Poisson propagation for deformable modelling , 2017, Neural Computing and Applications.

[79]  Lydia E. Kavraki,et al.  Planning Paths for a Flexible Surface Patch , 1998, Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No.98CH36146).

[80]  Kenneth Y. Goldberg,et al.  D-space and deform closure: a framework for holding deformable parts , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[81]  A. Nurnberger,et al.  A problem specific recurrent neural network for the description and simulation of dynamic spring models , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[82]  Franc Solina,et al.  Segmentation and Recovery of Superquadrics , 2000, Computational Imaging and Vision.

[83]  Ronen Basri,et al.  Determining the similarity of deformable shapes , 1998, Vision Research.

[84]  Andrea Cherubini,et al.  Model-free vision-based shaping of deformable plastic materials , 2020, Int. J. Robotics Res..

[85]  A. D. Marshall,et al.  Learnt Real‐time Meshless Simulation , 2014, Comput. Graph. Forum.

[86]  Konstantinos Moustakas,et al.  Adaptive representation of dynamic 3D meshes for low-latency applications , 2019, Comput. Aided Geom. Des..

[87]  Chris Maraffi.  Maya Character Creation: Modeling and Animation Controls , 2003 .

[88]  Mubarak Shah,et al.  A Fast algorithm for active contours and curvature estimation , 1992, CVGIP Image Underst..

[89]  John Kenneth Salisbury,et al.  Dynamic simulation of deformable objects using the Long Elements Method , 2002, Proceedings 10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. HAPTICS 2002.

[90]  Daniel Cremers,et al.  Dynamical statistical shape priors for level set-based tracking , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[91]  Demetri Terzopoulos,et al.  Deep Active Lesion Segmentation , 2019, bioRxiv.

[92]  Demetri Terzopoulos,et al.  Dynamically coupled particle systems for geometric modeling, reconstruction, and animation , 1998 .

[93]  Hamamache Kheddouci,et al.  New graph distance for deformable 3D objects recognition based on triangle-stars decomposition , 2019, Pattern Recognit..

[94]  Adriano N. Raposo,et al.  Pi-surfaces: products of implicit surfaces towards constructive composition of 3D objects , 2019, ArXiv.

[95]  Ahmad H. Nasri,et al.  T-splines and T-NURCCs , 2003, ACM Trans. Graph..

[96]  Dinesh Manocha,et al.  Path Planning for Deformable Robots in Complex Environments , 2005, Robotics: Science and Systems.

[97]  Timothy F. Cootes,et al.  Active Shape Models-Their Training and Application , 1995, Comput. Vis. Image Underst..

[98]  Shinichi Hirai,et al.  Robust grasping manipulation of deformable objects , 2001, Proceedings of the 2001 IEEE International Symposium on Assembly and Task Planning (ISATP2001). Assembly and Disassembly in the Twenty-first Century. (Cat. No.01TH8560).

[99]  Markus H. Gross,et al.  A versatile and robust model for geometrically complex deformable solids , 2004, Proceedings Computer Graphics International, 2004..

[100]  Miguel A. Otaduy,et al.  A Survey on Position‐Based Simulation Methods in Computer Graphics , 2014, Comput. Graph. Forum.

[101]  Michael Isard,et al.  Statistical models of visual shape and motion , 1998, Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[102]  Jiajun Wu,et al.  Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids , 2018, ICLR.

[103]  Dan Morris,et al.  Automatic preparation, calibration, and simulation of deformable objects , 2008, Computer methods in biomechanics and biomedical engineering.

[104]  Pierre Payeur,et al.  Multi-Modal Sensing and Robotic Manipulation of Non-Rigid Objects: A Survey , 2018, Robotics.

[105]  Marie-Paule Cani,et al.  An implicit formulation for precise contact modeling between flexible solids , 1993, SIGGRAPH.

[106]  Jernej Barbic,et al.  Real-Time subspace integration for St. Venant-Kirchhoff deformable models , 2005, ACM Trans. Graph..

[107]  B. Das,et al.  Inertial snake for contour detection in ultrasonography images , 2004 .

[108]  Leslie Pack Kaelbling,et al.  Augmenting Physical Simulators with Stochastic Neural Networks: Case Study of Planar Pushing and Bouncing , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[109]  Ana-Maria Cretu,et al.  Acquisition and Neural Network Prediction of 3D Deformable Object Shape Using a Kinect and a Force-Torque Sensor , 2017, Sensors.

[110]  Kenneth Y. Goldberg,et al.  Sensorless Motion Planning for Medical Needle Insertion in Deformable Tissues , 2009, IEEE Transactions on Information Technology in Biomedicine.

[111]  Dmitry Berenson,et al.  Manipulation of deformable objects without modeling and simulating deformation , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[112]  T. Barth,et al.  Finite Volume Methods: Foundation and Analysis , 2004 .

[113]  Hiroshi Sako,et al.  IMVIP 2007 International Machine Vision and Image Processing Conference , 2007 .

[114]  Allen Van Gelder,et al.  Approximate Simulation of Elastic Membranes by Triangulated Spring Meshes , 1998, J. Graphics, GPU, & Game Tools.

[115]  Marie-Paule Cani,et al.  Controlling Anisotropy in Mass-Spring Systems , 2000, Computer Animation and Simulation.

[116]  Christian Duriez,et al.  FEM-Based Deformation Control for Dexterous Manipulation of 3D Soft Objects , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[117]  Min Bai,et al.  Learning Deep Structured Active Contours End-to-End , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[118]  Timothy F. Cootes,et al.  Training Models of Shape from Sets of Examples , 1992, BMVC.

[119]  Yanhong Luo,et al.  Fusing force and vision feedback for manipulating deformable objects , 2001, J. Field Robotics.

[120]  Tae-Yong Kim,et al.  Unified particle physics for real-time applications , 2014, ACM Trans. Graph..

[121]  A OtaduyMiguel,et al.  A Survey on Position-Based Simulation Methods in Computer Graphics , 2014 .

[122]  Belhassen-Chedli Bouzgarrou,et al.  Robotic manipulation and sensing of deformable objects in domestic and industrial applications: a survey , 2018, Int. J. Robotics Res..

[123]  Alexandre Krupa,et al.  Tracking of Non-Rigid Objects using RGB-D Camera , 2019, 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC).

[124]  Markus H. Gross,et al.  Interactive Virtual Materials , 2004, Graphics Interface.

[125]  Alan L. Yuille,et al.  Feature extraction from faces using deformable templates , 2004, International Journal of Computer Vision.

[126]  Danica Kragic,et al.  Estimating deformability of objects using meshless shape matching , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[127]  Wolfgang Straßer,et al.  Analysis of numerical methods for the simulation of deformable models , 2003, The Visual Computer.

[128]  P. Jiménez,et al.  Survey on model-based manipulation planning of deformable objects , 2012 .

[129]  Javier Ruiz-del-Solar,et al.  Continuous perception for deformable objects understanding , 2019, Robotics Auton. Syst..

[130]  John C. Platt,et al.  Elastically deformable models , 1987, SIGGRAPH.

[131]  Yin Yang,et al.  Haptic-enabled interactive rendering of deformable objects based on shape matching , 2013, 2013 IEEE International Symposium on Haptic Audio Visual Environments and Games (HAVE).

[132]  David I. W. Levin,et al.  Latent‐space Dynamics for Reduced Deformable Simulation , 2019, Comput. Graph. Forum.

[133]  Hervé Delingette,et al.  General Object Reconstruction Based on Simplex Meshes , 1999, International Journal of Computer Vision.

[134]  Dinesh K. Pai,et al.  ArtDefo: accurate real time deformable objects , 1999, SIGGRAPH.

[135]  Doug L. James,et al.  Real time physics: class notes , 2008, SIGGRAPH '08.

[136]  Fernando Torres Medina,et al.  A tactile-based grasping strategy for deformable objects' manipulation and deformability estimation , 2015, 2015 12th International Conference on Informatics in Control, Automation and Robotics (ICINCO).