Fusing visual and tactile sensing for 3-D object reconstruction while grasping

In this work, we propose to reconstruct a complete 3-D model of an unknown object by fusion of visual and tactile information while the object is grasped. Assuming the object is symmetric, a first hypothesis of its complete 3-D shape is generated from a single view. This initial model is used to plan a grasp on the object which is then executed with a robotic manipulator equipped with tactile sensors. Given the detected contacts between the fingers and the object, the full object model including the symmetry parameters can be refined. This refined model will then allow the planning of more complex manipulation tasks. The main contribution of this work is an optimal estimation approach for the fusion of visual and tactile data applying the constraint of object symmetry. The fusion is formulated as a state estimation problem and solved with an iterative extended Kalman filter. The approach is validated experimentally using both artificial and real data from two different robotic platforms.

[1]  Tamim Asfour,et al.  A potential field approach to dexterous tactile exploration of unknown objects , 2008, Humanoids 2008 - 8th IEEE-RAS International Conference on Humanoid Robots.

[2]  Gary M. Bone,et al.  Automated modeling and robotic grasping of unknown three-dimensional objects , 2008, 2008 IEEE International Conference on Robotics and Automation.

[3]  Danica Kragic,et al.  Active 3D scene segmentation and detection of unknown objects , 2010, 2010 IEEE International Conference on Robotics and Automation.

[4]  Marc Toussaint,et al.  Gaussian process implicit surfaces for shape estimation and grasping , 2011, 2011 IEEE International Conference on Robotics and Automation.

[5]  Nico Blodow,et al.  Combined 2D–3D categorization and classification for multimodal perception systems , 2011, Int. J. Robotics Res..

[6]  Peter K. Allen,et al.  Integrating Vision and Touch for Object Recognition Tasks , 1988, Int. J. Robotics Res..

[7]  Jimmy A. Jørgensen,et al.  Grasping unknown objects using an Early Cognitive Vision system for general scene understanding , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[8]  Peter Allen.  Surface descriptions from vision and touch , 1984, ICRA.

[9]  Gabriel Taubin,et al.  Estimation of Planar Curves, Surfaces, and Nonplanar Space Curves Defined by Implicit Equations with Applications to Edge and Range Image Segmentation , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  Danica Kragic,et al.  Mind the gap - robotic grasping under incomplete observation , 2011, 2011 IEEE International Conference on Robotics and Automation.

[11]  Sebastian Thrun,et al.  Shape from symmetry , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[12]  Alberto Del Bimbo,et al.  Metric 3D reconstruction and texture acquisition of surfaces of revolution from a single uncalibrated view , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Takeo Kanade,et al.  Geometric reasoning for single image structure recovery , 2009, CVPR.

[14]  汤晓鸥,et al.  Symmetric Piecewise Planar Object Reconstruction from a Single Image , 2011 .

[15]  Ashutosh Saxena,et al.  Robotic Grasping of Novel Objects using Vision , 2008, Int. J. Robotics Res..

[16]  H. Opower.  Multiple view geometry in computer vision , 2002 .

[17]  B. Siciliano,et al.  Visual Grasp Planning for Unknown Objects Using a Multifingered Robotic Hand , 2013, IEEE/ASME Transactions on Mechatronics.

[18]  Jarmo Ilonen,et al.  Robust robot-camera calibration , 2011, 2011 15th International Conference on Advanced Robotics (ICAR).

[19]  Dan Simon,et al.  Optimal State Estimation: Kalman, H∞, and Nonlinear Approaches , 2006 .

[20]  Danijela Ristic-Durrant,et al.  Robust stereo-vision based 3D modelling of real-world objects for assistive robotic applications , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[21]  Yoshiki Uchikawa,et al.  A method of 3D object reconstruction by fusing vision with touch using internal models with global and local deformations , 1993, [1993] Proceedings IEEE International Conference on Robotics and Automation.

[22]  Ashutosh Saxena,et al.  Efficient grasping from RGBD images: Learning using a new rectangle representation , 2011, 2011 IEEE International Conference on Robotics and Automation.

[23]  Helge J. Ritter,et al.  A Probabilistic Approach to Tactile Shape Reconstruction , 2011, IEEE Transactions on Robotics.