Combined shape, appearance and silhouette for simultaneous manipulator and object tracking

This paper develops an estimation framework for sensor-guided manipulation of a rigid object via a robot arm. Using an unscented Kalman Filter (UKF), the method combines dense range information (from stereo cameras and 3D ranging sensors) as well as visual appearance features and silhouettes of the object and manipulator to track both an object-fixed frame location as well as a manipulator tool or palm frame location. If available, tactile data is also incorporated. By using these different imaging sensors and different imaging properties, we can leverage the advantages of each sensor and each feature type to realize more accurate and robust object and reference frame tracking. The method is demonstrated using the DARPA ARM-S system, consisting of a Barrett™WAM manipulator.

[1]  I ROCK,et al.  Vision and Touch: An Experimentally Created Conflict between the Two Senses , 1964, Science.

[2]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Jeffrey K. Uhlmann,et al.  New extension of the Kalman filter to nonlinear systems , 1997, Defense, Security, and Sensing.

[4]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[5]  José Manuel Iñesta Quereda,et al.  Vision-guided grasping of unknown objects for service robots , 1998, Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No.98CH36146).

[6]  Olivier D. Faugeras,et al.  3D articulated models and multi-view tracking with silhouettes , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[7]  Danica Kragic,et al.  Real-time tracking meets online grasp planning , 2001, Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164).

[8]  S. Tosovic,et al.  On Combining Shape from Silhouette and Shape from Structured Light ∗ , 2002 .

[9]  Stan Sclaroff,et al.  Estimating 3D hand pose from a cluttered image , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[10]  Francis Schmitt,et al.  Silhouette and stereo fusion for 3D object modeling , 2003, Fourth International Conference on 3-D Digital Imaging and Modeling, 2003. 3DIM 2003. Proceedings..

[11]  Michael I. Mandel,et al.  Visual Hand Tracking Using Nonparametric Belief Propagation , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[12]  Nicholas R. Howe,et al.  Silhouette Lookup for Automatic Pose Tracking , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[13]  Donald B. Gennery,et al.  Generalized Camera Calibration Including Fish-Eye Lenses , 2006, International Journal of Computer Vision.

[14]  Paul Backes,et al.  Visual end‐effector position error compensation for planetary robotics , 2007, J. Field Robotics.

[15]  Konrad Schindler,et al.  A Generalisation of the ICP Algorithm for Articulated Bodies , 2008, BMVC.

[16]  Jana Kosecka,et al.  Visual door detection integrating appearance and shape cues , 2008, Robotics Auton. Syst..

[17]  Ashutosh Saxena,et al.  Robotic Grasping of Novel Objects using Vision , 2008, Int. J. Robotics Res..

[18]  John Kenneth Salisbury,et al.  Vision based 3-D shape sensing of flexible manipulators , 2008, 2008 IEEE International Conference on Robotics and Automation.

[19]  Jitendra Malik,et al.  Shape matching and object recognition using shape contexts , 2010, 2010 3rd International Conference on Computer Science and Information Technology.

[20]  Carl E. Rasmussen,et al.  Learning to Control a Low-Cost Manipulator using Data-Efficient Reinforcement Learning , 2011, Robotics: Science and Systems.

[21]  Tim D. Barfoot,et al.  A Serial Approach to Handling High-Dimensional Measurements in the Sigma-Point Kalman Filter , 2011, Robotics: Science and Systems.

[22]  Joel W. Burdick,et al.  Fusion of stereo vision, force-torque, and joint sensors for estimation of in-hand object location , 2011, 2011 IEEE International Conference on Robotics and Automation.

[23]  Andrew Y. Ng,et al.  A low-cost compliant 7-DOF robotic manipulator , 2011, 2011 IEEE International Conference on Robotics and Automation.

[24]  Dieter Fox,et al.  Manipulator and object tracking for in-hand 3D object modeling , 2011, Int. J. Robotics Res..