Robot-to-human feedback and automatic object grasping using an RGB-D camera–projector system

This paper presents a novel system for human–robot interaction in object-grasping applications. Consisting of an RGB-D camera, a projector and a robot manipulator, the proposed system provides intuitive information to the human by analyzing the scene, detecting graspable objects and directly projecting numbers or symbols in front of objects. Objects are detected using a visual attention model that incorporates color, shape and depth information. The positions and orientations of the projected numbers are based on the shapes, positions and orientations of the corresponding objects. Users select a grasping target by indicating the corresponding number. Projected arrows are then created on the fly to guide a robotic arm to grasp the selected object using visual servoing and deliver the object to the human user. Experimental results are presented to demonstrate how the system is used in robot grasping tasks.

[1]  Tzou Jyh-Hwa,et al.  The development of the restaurant service mobile robot with a Laser positioning system , 2008, 2008 27th Chinese Control Conference.

[2]  Advait Jain,et al.  A clickable world: Behavior selection through pointing and context for mobile manipulation , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[3]  J. W. Minett,et al.  Optimizing the P300-based brain–computer interface: current status, limitations and future directions , 2011, Journal of neural engineering.

[4]  Bo Han,et al.  High Speed Visual Saliency Computation on GPU , 2007, 2007 IEEE International Conference on Image Processing.

[5]  Josef Kittler,et al.  A survey of the hough transform , 1988, Comput. Vis. Graph. Image Process..

[6]  Helge J. Ritter,et al.  Integrating Context-Free and Context-Dependent Attentional Mechanisms for Gestural Object Reference , 2003, ICVS.

[7]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[8]  Ashutosh Saxena,et al.  Robotic Grasping of Novel Objects using Vision , 2008, Int. J. Robotics Res..

[9]  Joel W. Burdick,et al.  A probabilistic framework for stereo-vision based 3D object search with 6D pose estimation , 2010, 2010 IEEE International Conference on Robotics and Automation.

[10]  Andrej Gams,et al.  Real-time full body motion imitation on the COMAN humanoid robot , 2014, Robotica.

[11]  François Chaumette,et al.  2 1/2 D Visual Servoing with Respect to Unknown Objects Through a New Estimation Scheme of Camera Displacement , 2000, International Journal of Computer Vision.

[12]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[13]  Shinichiro Omachi,et al.  Fast Template Matching With Polynomials , 2007, IEEE Transactions on Image Processing.

[14]  Stefan Schaal,et al.  Overt visual attention for a humanoid robot , 2001, Proceedings 2001 IEEE/RSJ International Conference on Intelligent Robots and Systems. Expanding the Societal Role of Robotics in the the Next Millennium (Cat. No.01CH37180).

[15]  Nicholas R. Gans,et al.  Stable Visual Servoing Through Hybrid Switched-System Control , 2007, IEEE Transactions on Robotics.

[16]  Guoqiang Hu,et al.  Adaptive visual servo control to simultaneously stabilize image and pose error , 2012 .

[17]  William J. Wilson,et al.  Hybrid motion control and planning strategies for visual servoing , 2005, IEEE Transactions on Industrial Electronics.

[18]  Claudio S. Pinhanez,et al.  Interacting with steerable projected displays , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[19]  Joaquim Salvi,et al.  Pattern codification strategies in structured light systems , 2004, Pattern Recognit..

[20]  Kimitoshi Yamazaki,et al.  Home-Assistant Robot for an Aging Society , 2012, Proceedings of the IEEE.

[21]  S. Shankar Sastry,et al.  An Invitation to 3-D Vision , 2004 .

[22]  Nicholas R. Gans,et al.  A trifocal tensor based camera-projector system for robot-human interaction , 2012, 2012 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[23]  Amit Kumar Pandey,et al.  Towards planning Human-Robot Interactive manipulation tasks: Task dependent and human oriented autonomous selection of grasp and placement , 2012, 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob).

[24]  Dieter Fox,et al.  Depth kernel descriptors for object recognition , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[25]  Danica Kragic,et al.  An Active Vision System for Detecting, Fixating and Manipulating Objects in the Real World , 2010, Int. J. Robotics Res..

[26]  Guoqiang Hu,et al.  Homography-Based Visual Servo Control With Imperfect Camera Calibration , 2009, IEEE Transactions on Automatic Control.

[27]  S. Yamaguchi,et al.  Kinect cane: An assistive system for the visually impaired based on three-dimensional object recognition , 2012, 2012 IEEE/SICE International Symposium on System Integration (SII).

[28]  Paul A. Beardsley,et al.  Interaction using a handheld projector , 2005, IEEE Computer Graphics and Applications.

[29]  Danica Kragic,et al.  Attention-based active 3D point cloud segmentation , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[30]  Christophe Collewet,et al.  A Camera-Projector System for Robot Positioning by Visual Servoing , 2006, 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'06).

[31]  Simone Frintrop,et al.  Attentional Landmarks and Active Gaze Control for Visual SLAM , 2008, IEEE Transactions on Robotics.

[32]  Akinori Ito,et al.  ASAHI: OK for failure A robot for supporting daily life, equipped with a robot avatar , 2013, 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[33]  Nicolas D. Georganas,et al.  Real-Time Hand Gesture Detection and Recognition Using Bag-of-Features and Support Vector Machine Techniques , 2011, IEEE Transactions on Instrumentation and Measurement.

[34]  Chong Ho Lee,et al.  Interactive display robot: Projector robot with natural user interface , 2013, 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[35]  Jianpeng Zhou,et al.  Real Time Robust Human Detection and Tracking System , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[36]  Nicholas R. Gans,et al.  A Multi-view camera-projector system for object detection and robot-human feedback , 2013, 2013 IEEE International Conference on Robotics and Automation.

[37]  François Chaumette,et al.  Visual servo control. I. Basic approaches , 2006, IEEE Robotics & Automation Magazine.