Bi-Manual Articulated Robot Teleoperation using an External RGB-D Range Sensor

In this paper, we present an implementation of a bi-manual teleoperation system, controlled by a human through three-dimensional (3D) skeleton extraction. The input data is given from a cheap RGB-D range sensor, such as the ASUS Xtion PRO. To achieve this, we have implemented a 3D version of the impressive OpenPose package, which was recently developed. The first stage of our method contains the execution of the OpenPose Convolutional Neural Network (CNN), using a sequence of RGB images as input. The extracted human skeleton pose localisation in two-dimensions (2D) is followed by the mapping of the extracted joint location estimations into their 3D pose in the camera frame. The output of this process is then used as input to drive the end-pose of the robotic hands relative to the human hand movements, through a whole-body inverse kinematics process in the Cartesian space. Finally, we implement the method as a ROS wrapper package and we test it on the centaur-like CENTAURO robot. Our demonstrated task is of a box and lever manipulation in real-time, as a result of a human task demonstration.

[1]  Maja J. Mataric,et al.  Motion capture from inertial sensing for untethered humanoid teleoperation , 2004, 4th IEEE/RAS International Conference on Humanoid Robots, 2004..

[2]  Varun Ramakrishna,et al.  Convolutional Pose Machines , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Jan Peters,et al.  First-person tele-operation of a humanoid robot , 2015, 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids).

[4]  Aloysius K. Mok,et al.  Web Based Teleoperation of a Humanoid Robot , 2016, ArXiv.

[5]  Mattia Poggiani,et al.  Humanoids at Work: The WALK-MAN Robot in a Postearthquake Scenario , 2018, IEEE Robotics & Automation Magazine.

[6]  Sven Behnke,et al.  Intuitive bimanual telemanipulation under communication restrictions by immersive 3D visualization and motion tracking , 2015, 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids).

[7]  Jörn Malzahn,et al.  Development of a human size and strength compliant bi-manual platform for realistic heavy manipulation tasks , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[8]  C. Stanton,et al.  Teleoperation of a humanoid robot using full-body motion capture , example movements , and machine learning , 2012 .

[9]  Nikolaos G. Tsagarakis,et al.  Robot control for dummies: Insights and examples using OpenSoT , 2017, 2017 IEEE-RAS 17th International Conference on Humanoid Robotics (Humanoids).

[10]  Bernt Schiele,et al.  DeeperCut: A Deeper, Stronger, and Faster Multi-person Pose Estimation Model , 2016, ECCV.

[11]  Satoshi Kagami,et al.  Humanoid teleoperation for whole body manipulation , 2008, 2008 IEEE International Conference on Robotics and Automation.

[12]  Takeo Kanade,et al.  Panoptic Studio: A Massively Multiview System for Social Interaction Capture , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Vincent Lepetit,et al.  DeepPrior++: Improving Fast and Accurate 3D Hand Pose Estimation , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[14]  Ennio Gambi,et al.  A Human Activity Recognition System Using Skeleton Data from RGBD Sensors , 2016, Comput. Intell. Neurosci..

[15]  Song Yang,et al.  Teleoperation Humanoid Robot Control System Based on Kinect Sensor , 2012, 2012 4th International Conference on Intelligent Human-Machine Systems and Cybernetics.

[16]  Andreas Zell,et al.  A Comparison of 3D Sensors for Wheeled Mobile Robots , 2014, IAS.

[17]  S Stefan Lichiardopol,et al.  A Survey on Teleoperation , 2007 .

[18]  Nikolaos G. Tsagarakis,et al.  A Self-Tuning Impedance Controller for Autonomous Robotic Manipulation , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[19]  Darwin G. Caldwell,et al.  Learning Competing Constraints and Task Priorities from Demonstrations of Bimanual Skills , 2017, ArXiv.

[20]  Yi Yang,et al.  Articulated Human Detection with Flexible Mixtures of Parts , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Darwin G. Caldwell,et al.  Transfer learning of shared latent spaces between robots with similar kinematic structure , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[22]  Nikolaos G. Tsagarakis,et al.  An affordance-based pilot interface for high-level control of humanoid robots in supervised autonomy , 2016, 2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids).

[23]  Nikolaos G. Tsagarakis,et al.  HEXOTRAC: A highly under-actuated hand exoskeleton for finger tracking and force feedback , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[24]  Michael A. Goodrich,et al.  Teleoperation and Beyond for Assistive Humanoid Robots , 2013 .

[25]  Dmitry Berenson,et al.  Toward a user-guided manipulation framework for high-DOF robots with limited communication , 2013, 2013 IEEE Conference on Technologies for Practical Robot Applications (TePRA).

[26]  Emanuele Menegatti,et al.  Teaching humanoid robotics by means of human teleoperation through RGB-D sensors , 2016, Robotics Auton. Syst..

[27]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Magnus Egerstedt,et al.  Humanoid robot teleoperation for tasks with power tools , 2013, 2013 IEEE Conference on Technologies for Practical Robot Applications (TePRA).

[29]  Sergio Escalera,et al.  RGB-D-based Human Motion Recognition with Deep Learning: A Survey , 2017, Comput. Vis. Image Underst..

[30]  James J. Little,et al.  A Simple Yet Effective Baseline for 3d Human Pose Estimation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[31]  Takeshi Oishi,et al.  Motion generation of the humanoid robot for teleoperation by task model , 2015, 2015 24th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN).

[32]  Nikolaos G. Tsagarakis,et al.  XBotCore: A Real-Time Cross-Robot Software Platform , 2017, 2017 First IEEE International Conference on Robotic Computing (IRC).

[33]  Angelo Cangelosi,et al.  Development of Kinect based teleoperation of Nao robot , 2016, 2016 International Conference on Advanced Robotics and Mechatronics (ICARM).

[34]  Nikolaos G. Tsagarakis,et al.  Center-of-Mass-Based Grasp Pose Adaptation Using 3D Range and Force/Torque Sensing , 2018, Int. J. Humanoid Robotics.