RoboCoDraw: Robotic Avatar Drawing with GAN-based Style Transfer and Time-efficient Path Optimization

Robotic drawing has become increasingly popular as an entertainment and interactive tool. In this paper we present RoboCoDraw, a real-time collaborative robot-based drawing system that draws stylized human face sketches interactively in front of human users, by using the Generative Adversarial Network (GAN)-based style transfer and a Random-Key Genetic Algorithm (RKGA)-based path optimization. The proposed RoboCoDraw system takes a real human face image as input, converts it to a stylized avatar, then draws it with a robotic arm. A core component in this system is the Avatar-GAN proposed by us, which generates a cartoon avatar face image from a real human face. AvatarGAN is trained with unpaired face and avatar images only and can generate avatar images of much better likeness with human face images in comparison with the vanilla CycleGAN. After the avatar image is generated, it is fed to a line extraction algorithm and converted to sketches. An RKGA-based path optimization algorithm is applied to find a time-efficient robotic drawing path to be executed by the robotic arm. We demonstrate the capability of RoboCoDraw on various face images using a lightweight, safe collaborative robot UR5.

[1]  Craig S. Kaplan.  TSP Art , 2005 .

[2]  Chuan Li,et al.  Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks , 2016, ECCV.

[3]  Seungyong Lee,et al.  Flow-Based Image Abstraction , 2009, IEEE Transactions on Visualization and Computer Graphics.

[4]  Pierre Dillenbourg,et al.  When Children Teach a Robot to Write: An Autonomous Teachable Humanoid Which Uses Simulated Handwriting , 2015, 2015 10th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[5]  Joshua Correll,et al.  The Chicago face database: A free stimulus set of faces and norming data , 2015, Behavior research methods.

[6]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  M. Marchese,et al.  An ant colony optimization method for generalized TSP problem , 2008 .

[9]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[10]  Andrea Vedaldi,et al.  Instance Normalization: The Missing Ingredient for Fast Stylization , 2016, ArXiv.

[11]  Young J. Kim,et al.  Artistic Pen Drawing on an Arbitrary Surface Using an Impedance-Controlled Robot , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[12]  Amit R.Sharma,et al.  Face Photo-Sketch Synthesis and Recognition , 2012 .

[13]  Jing Liao,et al.  CariGANs , 2018, ACM Trans. Graph..

[14]  Maria Luiza Recena Menezes,et al.  Design for an Art Therapy Robot: An Explorative Review of the Theoretical Foundations for Engaging in Emotional and Creative Painting with a Robot , 2018, Multimodal Technologies and Interaction.

[15]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Tat-Seng Chua,et al.  Laplacian-Steered Neural Style Transfer , 2017, ACM Multimedia.

[17]  G. Jean-Pierre,et al.  The artist robot: A robot drawing like a human artist , 2012, 2012 IEEE International Conference on Industrial Technology.

[18]  Nikos A. Aspragathos,et al.  Optimal robot task scheduling based on genetic algorithms , 2005 .

[19]  James C. Bean,et al.  Genetic Algorithms and Random Keys for Sequencing and Optimization , 1994, INFORMS J. Comput..

[20]  Frank Ortmeier,et al.  Robotic Task Sequencing Problem: A Survey , 2015, Journal of Intelligent & Robotic Systems.

[21]  Brian W. Kernighan,et al.  An Effective Heuristic Algorithm for the Traveling-Salesman Problem , 1973, Oper. Res..

[22]  Aude Billard,et al.  A humanoid robot drawing human portraits , 2005, 5th IEEE-RAS International Conference on Humanoid Robots, 2005..

[23]  J. C. Bean,et al.  An efficient transformation of the generalized traveling salesman problem , 1993 .

[24]  Chyi-Yeu Lin,et al.  Human portrait generation system for robot arm drawing , 2009, 2009 IEEE/ASME International Conference on Advanced Intelligent Mechatronics.

[25]  Frederic Fol Leymarie,et al.  Portrait drawing by Paul the robot , 2013, Comput. Graph..

[26]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[27]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.