Learning to gesticulate by observation using a deep generative approach

The goal of the system presented in this paper is to develop a natural talking gesture generation behavior for a humanoid robot, by feeding a Generative Adversarial Network (GAN) with human talking gestures recorded by a Kinect. A direct kinematic approach is used to translate from human poses to robot joint positions. The provided videos show that the robot is able to use a wide variety of gestures, offering a non-dreary, natural expression level.

[1]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  José Manuel Gutiérrez,et al.  Learning Bayesian Networks , 1997 .

[3]  Frank Chongwoo Park,et al.  Using Hidden Markov Models to Generate Natural Humanoid Movement , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[4]  Hans-Peter Seidel,et al.  VNect , 2017, ACM Trans. Graph..

[5]  Silvio Savarese,et al.  Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[6]  Itziar Irigoien,et al.  Spontaneous talking gestures using Generative Adversarial Networks , 2019, Robotics Auton. Syst..

[7]  Majid Nili Ahmadabadi,et al.  Inverse Kinematics Based Human Mimicking System using Skeletal Tracking Technology , 2017, J. Intell. Robotic Syst..

[8]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[9]  Ajay Kumar Tanwani,et al.  Generative Models for Learning Robot Manipulation Skills from Humans , 2018 .

[10]  Nadia Magnenat-Thalmann,et al.  Body Movements Generation for Virtual Characters and Social Robots , 2017, Social Signal Processing.

[11]  Wolfram Burgard,et al.  Automatic bone parameter estimation for skeleton tracking in optical motion capture , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[12]  Shuyang Lin,et al.  Real-Time Whole-Body Imitation by Humanoid Robots and Task-Oriented Teleoperation Using an Analytical Mapping Method and Quantitative Evaluation , 2018, Applied Sciences.

[13]  Sophie Sakka,et al.  Support changes during online human motion imitation by a humanoid robot using task specification , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[14]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[15]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[16]  Elena Lazkano,et al.  Singing minstrel robots, a means for improving social behaviors , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[17]  Katsushi Ikeuchi,et al.  Toward a Dancing Robot With Listening Capability: Keypose-Based Integration of Lower-, Middle-, and Upper-Body Motions for Varying Music Tempos , 2014, IEEE Transactions on Robotics.

[18]  Santosha K. Dwivedy,et al.  Inverse kinematics of a NAO humanoid robot using kinect to track and imitate human motion , 2015, 2015 International Conference on Robotics, Automation, Control and Embedded Systems (RACE).

[19]  Giovanni Pilato,et al.  Creative Robot Dance with Variational Encoder , 2017, ICCC.

[20]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Nicholette D. Palmer,et al.  Novel genetic associations for blood pressure identified via gene-alcohol interaction in up to 570K individuals across multiple ancestries , 2018, PloS one.

[22]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[23]  Salvatore Gaglio,et al.  An automatic system for humanoid dance creation , 2016, BICA 2016.

[24]  Carmadi Machbub,et al.  Human gesture imitation on NAO humanoid robot using kinect based on inverse kinematics method , 2017, 2017 International Conference on Advanced Mechatronics, Intelligent Manufacture, and Industrial Automation (ICAMIMIA).

[25]  Ian J. Goodfellow,et al.  NIPS 2016 Tutorial: Generative Adversarial Networks , 2016, ArXiv.

[26]  Thierry Dutoit,et al.  Robust and automatic motion-capture data recovery using soft skeleton constraints and model averaging , 2018, PloS one.

[27]  Paul Geladi,et al.  Principal Component Analysis , 1987, Comprehensive Chemometrics.

[28]  Elena Lazkano,et al.  Humanizing NAO robot teleoperation using ROS , 2014, 2014 IEEE-RAS International Conference on Humanoid Robots.

[29]  C. Creider.  Hand and Mind: What Gestures Reveal about Thought , 1994 .

[30]  Cynthia Breazeal,et al.  Designing sociable robots , 2002 .

[31]  B. Everitt,et al.  Finite Mixture Distributions , 1981 .