Manipulator and object tracking for in-hand 3D object modeling

Recognizing and manipulating objects is an important task for mobile robots performing useful services in everyday environments. While existing techniques for object recognition related to manipulation provide very good results even for noisy and incomplete data, they are typically trained using data generated in an offline process. As a result, they do not enable a robot to acquire new object models as it operates in an environment. In this paper we develop an approach to building 3D models of unknown objects based on a depth camera observing the robot’s hand while moving an object. The approach integrates both shape and appearance information into an articulated Iterative Closest Point approach to track the robot’s manipulator and the object. Objects are modeled by sets of surfels, which are small patches providing occlusion and appearance information. Experiments show that our approach provides very good 3D models even when the object is highly symmetric and lacks visual features and the manipulator motion is noisy. Autonomous object modeling represents a step toward improved semantic understanding, which will eventually enable robots to reason about their environments in terms of objects and their relations rather than through raw sensor data.

[1]  Gérard G. Medioni,et al.  Object modelling by registration of multiple range images , 1992, Image Vis. Comput..

[2]  Marc Levoy,et al.  Zippered polygon meshes from range images , 1994, SIGGRAPH.

[3]  Marc Levoy,et al.  A volumetric method for building complex models from range images , 1996, SIGGRAPH.

[4]  Katsushi Ikeuchi,et al.  Object shape and reflectance modeling from observation , 1997, SIGGRAPH.

[5]  Andrew E. Johnson,et al.  Spin-Images: A Representation for 3-D Surface Matching , 1997 .

[6]  Andrew E. Johnson,et al.  Registration and integration of textured 3-D data , 1997, Proceedings. International Conference on Recent Advances in 3-D Digital Imaging and Modeling (Cat. No.97TB100134).

[7]  Sebastian Weik,et al.  Registration of 3-D partial surface models using luminance and depth information , 1997, Proceedings. International Conference on Recent Advances in 3-D Digital Imaging and Modeling (Cat. No.97TB100134).

[8]  Marc Levoy,et al.  The digital Michelangelo project: 3D scanning of large statues , 2000, SIGGRAPH.

[9]  Matthias Zwicker,et al.  Surfels: surface elements as rendering primitives , 2000, SIGGRAPH.

[10]  Robert Bergevin,et al.  A method for the registration of attributed range images , 2001, Proceedings Third International Conference on 3-D Digital Imaging and Modeling.

[11]  Sang Wook Lee,et al.  ICP Registration Using Invariant Features , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  Marc Levoy,et al.  Real-time 3D model acquisition , 2002, ACM Trans. Graph..

[13]  Stan Sclaroff,et al.  Estimating 3D hand pose from a cluttered image , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[14]  W. Burgard,et al.  First steps towards a robotic system for flexible volumetric mapping of indoor environments , 2004, IFAC Proceedings Volumes.

[15]  Michael I. Mandel,et al.  Visual Hand Tracking Using Nonparametric Belief Propagation , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[16]  Peter K. Allen,et al.  Graspit! A versatile simulator for robotic grasping , 2004, IEEE Robotics & Automation Magazine.

[17]  Doug L. James,et al.  Scanning physical interaction behavior of 3D objects , 2001, SIGGRAPH Courses.

[18]  Miguel Arias-Estrada,et al.  Iterative Closest SIFT Formulation for Robust Feature Matching , 2006, ISVC.

[19]  Michael M. Kazhdan,et al.  Poisson surface reconstruction , 2006, SGP '06.

[20]  S. Druon,et al.  Color Constrained ICP for Registration of Large Unstructured 3D Color Data Sets , 2006, 2006 IEEE International Conference on Information Acquisition.

[21]  Matei T. Ciocarlie,et al.  Dimensionality reduction for hand-independent dexterous robotic grasping , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[22]  Stefano Corazza,et al.  Accurately measuring human movement using articulated ICP with soft-joint constraints and a repository of articulated models , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[23]  Changchang Wu,et al.  SiftGPU : A GPU Implementation of Scale Invariant Feature Transform (SIFT) , 2007 .

[24]  Wolfram Burgard,et al.  Efficient estimation of accurate maximum likelihood maps in 3D , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[25]  Leif Kobbelt,et al.  A Surface-Growing Approach to Multi-View Stereo Reconstruction , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Gordon Cheng,et al.  Making Object Learning and Recognition an Active Process , 2008, Int. J. Humanoid Robotics.

[27]  Konrad Schindler,et al.  A Generalisation of the ICP Algorithm for Articulated Bodies , 2008, BMVC.

[28]  Danica Kragic,et al.  Birth of the Object: Detection of Objectness and Extraction of Object Shape through Object-Action complexes , 2008, Int. J. Humanoid Robotics.

[29]  Ashutosh Saxena,et al.  Robotic Grasping of Novel Objects using Vision , 2008, Int. J. Robotics Res..

[30]  Siddhartha S. Srinivasa,et al.  Grasp synthesis in cluttered environments for dexterous hands , 2008, Humanoids 2008 - 8th IEEE-RAS International Conference on Humanoid Robots.

[31]  Tom Drummond,et al.  ProFORMA: Probabilistic Feature-based On-line Rapid Model Acquisition , 2009, BMVC.

[32]  Dieter Fox,et al.  3D laser scan classification using web data and domain adaptation , 2009, Robotics: Science and Systems.

[33]  Tamim Asfour,et al.  Stereo-Based vs. Monocular 6-DoF Pose Estimation Using Point Features: A Quantitative Comparison , 2009, AMS.

[34]  Nicholas Roy,et al.  Probabilistic Models of Object Geometry with Application to Grasping , 2009, Int. J. Robotics Res..

[35]  Aram Kawewong,et al.  Wide-Baseline Visible Features for Highly Dynamic Scene Recognition , 2009, CAIP.

[36]  Siddhartha S. Srinivasa,et al.  Object recognition and full pose registration from a single image for robotic manipulation , 2009, 2009 IEEE International Conference on Robotics and Automation.

[37]  Luc Van Gool,et al.  In-hand scanning with online loop closure , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[38]  Alexander Verl,et al.  6D object localization and obstacle detection for collision-free manipulation with a mobile service robot , 2009, 2009 International Conference on Advanced Robotics.

[39]  Darius Burschka,et al.  The self-referenced DLR 3D-modeler , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[40]  Wai Ho Li,et al.  Interactive learning of visually symmetric objects , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[41]  Wolfram Burgard,et al.  Body schema learning for robotic manipulators from visual self-perception , 2009, Journal of Physiology - Paris.

[42]  Danica Kragic,et al.  An Active Vision System for Detecting, Fixating and Manipulating Objects in the Real World , 2010, Int. J. Robotics Res..

[43]  Ling Xu,et al.  Human-guided grasp measures improve grasp robustness on physical robot , 2010, 2010 IEEE International Conference on Robotics and Automation.

[44]  Takeo Kanade,et al.  Automated Construction of Robotic Manipulation Programs , 2010 .

[45]  Dieter Fox,et al.  RGB-D Mapping: Using Depth Cameras for Dense 3D Modeling of Indoor Environments , 2010, ISER.

[46]  W. S. Owen,et al.  Laser Scanner-based End-effector Tracking and Joint Variable Extraction for Heavy Machinery , 2010, Int. J. Robotics Res..

[47]  Sebastian Thrun,et al.  Real time motion capture using a single time-of-flight camera , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[48]  Dieter Fox,et al.  Autonomous generation of complete 3D object models using next best view manipulation planning , 2011, 2011 IEEE International Conference on Robotics and Automation.