Programming by Demonstration with User-Specified Perceptual Landmarks

Programming by demonstration (PbD) is an effective technique for developing complex robot manipulation tasks, such as opening bottles or using human tools. In order for such tasks to generalize to new scenes, the robot needs to be able to perceive objects, object parts, or other task-relevant parts of the scene. Previous work has relied on rigid, task-specific perception systems for this purpose. This paper presents a flexible and open-ended perception system that lets users specify perceptual "landmarks" during the demonstration, by capturing parts of the point cloud from the demonstration scene. We present a method for localizing landmarks in new scenes and experimentally evaluate this method in a variety of settings. Then, we provide examples where user-specified landmarks are used together with PbD on a PR2 robot to perform several complex manipulation tasks. Finally, we present findings from a user evaluation of our landmark specification interface demonstrating its feasibility as an end-user tool.

[1]  Rüdiger Dillmann,et al.  Incremental Learning of Tasks From User Demonstrations, Past Experiences, and Vocal Comments , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[2]  Jun Nakanishi,et al.  Learning Movement Primitives , 2005, ISRR.

[3]  Brett Browning,et al.  A survey of robot learning from demonstration , 2009, Robotics Auton. Syst..

[4]  Sonia Chernova,et al.  A Practical Comparison of Three Robot Learning from Demonstration Algorithm , 2012, Int. J. Soc. Robotics.

[5]  Stefan Schaal,et al.  Learning and generalization of motor skills by learning from demonstration , 2009, 2009 IEEE International Conference on Robotics and Automation.

[6]  Stefan Schaal,et al.  Robot Learning From Demonstration , 1997, ICML.

[7]  Gary Anthes,et al.  Deep learning comes of age , 2013, CACM.

[8]  Sethu Vijayakumar,et al.  ICRA '09. IEEE International Conference on Robotics and Automation, 2009 , 2009 .

[9]  Aude Billard,et al.  Statistical Learning by Imitation of Competing Constraints in Joint Space and Task Space , 2009, Adv. Robotics.

[10]  Yu Zhao,et al.  Robot learning from human demonstration with remote lead hrough teaching , 2016, 2016 European Control Conference (ECC).

[11]  Andrea Lockerd Thomaz,et al.  Simultaneously learning actions and goals from demonstration , 2016, Auton. Robots.

[12]  Henrik Gordon Petersen,et al.  Pose estimation using local structure-specific shape and appearance context , 2013, 2013 IEEE International Conference on Robotics and Automation.

[13]  Jianxiong Xiao,et al.  3D ShapeNets: A deep representation for volumetric shapes , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Maya Cakmak,et al.  Robot Programming by Demonstration with Interactive Action Visualizations , 2014, Robotics: Science and Systems.

[15]  Scott Niekum,et al.  Learning and generalization of complex tasks from unstructured demonstrations , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[16]  Nico Blodow,et al.  Fast Point Feature Histograms (FPFH) for 3D registration , 2009, 2009 IEEE International Conference on Robotics and Automation.

[17]  Maya Cakmak,et al.  Enhanced robotic cleaning with a low-cost tool attachment , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[18]  Maya Cakmak,et al.  Trajectories and keyframes for kinesthetic teaching: A human-robot interaction perspective , 2012, 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[19]  Andrea Lockerd Thomaz,et al.  Robot Learning from Human Teachers , 2014, Robot Learning from Human Teachers.

[20]  Rüdiger Dillmann,et al.  A comparison of four fast vision based object recognition methods for programming by demonstration applications , 2000, Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065).

[21]  Sebastian Scherer,et al.  VoxNet: A 3D Convolutional Neural Network for real-time object recognition , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[22]  Tamim Asfour,et al.  Action sequence reproduction based on automatic segmentation and Object-Action Complexes , 2013, 2013 13th IEEE-RAS International Conference on Humanoid Robots (Humanoids).

[23]  Stefan Schaal,et al.  Robot Programming by Demonstration , 2009, Springer Handbook of Robotics.

[24]  Scott Niekum,et al.  Incremental Semantically Grounded Learning from Demonstration , 2013, Robotics: Science and Systems.

[25]  Pieter Abbeel,et al.  Learning from Demonstrations Through the Use of Non-rigid Registration , 2013, ISRR.

[26]  Paul J. Besl,et al.  Method for registration of 3-D shapes , 1992, Other Conferences.

[27]  Danica Kragic,et al.  Robot Learning from Demonstration: A Task-level Planning Approach , 2008 .