Robot Learning from Demonstration Based on Action and Object Recognition

In this paper, we propose a vision-based robot learning from demonstration (LfD) system, using action and object recognition, to allow a robot to learn the behavior from a human demonstrator. The vision-based robot LfD system employs RGB cameras as sensing devices, and integrates object detection achieved by YOLO deep learning architecture and action recognition carried out by an I3D deep learning network to separate the overall demonstration into a few sub-actions. Finally, the corresponding robot moving trajectories are subsequently planned such that the actions can be performed successfully. KeywordsLearning from Demonstration, Action Recognition, YOLO, I3D, Mimic Robot, Deep Learning.

[1]  Stefan Schaal,et al.  Robot Programming by Demonstration , 2009, Springer Handbook of Robotics.

[2]  Brett Browning,et al.  A survey of robot learning from demonstration , 2009, Robotics Auton. Syst..

[3]  Maya Cakmak,et al.  Trajectories and keyframes for kinesthetic teaching: A human-robot interaction perspective , 2012, 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[4]  Dorthe Sølvason,et al.  Teleoperation for learning by demonstration: Data glove versus object manipulation for intuitive robot control , 2014, 2014 6th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops (ICUMT).

[5]  Maya Cakmak,et al.  Robot Programming by Demonstration with Interactive Action Visualizations , 2014, Robotics: Science and Systems.

[6]  Ales Ude,et al.  Solving peg-in-hole tasks by human demonstration and exception strategies , 2014 .

[7]  Andrew Zisserman,et al.  Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Feng Gao,et al.  Feeling the force: Integrating force and pose for fluent discovery through imitation learning to open medicine bottles , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[9]  Hongxin Zhang,et al.  Recent Advances on Vision-Based Robot Learning by Demonstration , 2018 .

[10]  Nikolaos G. Tsagarakis,et al.  Translating Videos to Commands for Robotic Manipulation with Deep Recurrent Neural Networks , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[11]  Wei-Yen Wang,et al.  Development of a Mimic Robot: Learning from Human Demonstration to Manipulate a Coffee Maker as an Example , 2019, 2019 IEEE 23rd International Symposium on Consumer Technologies (ISCT).