A Virtual Makeup Augmented Reality System

Virtual makeup systems allow users to try makeup on remotely, without wasting products or spending time cleaning up later. Some virtual makeup systems just show the final makeup result on the user's face, while other systems allow some interaction when applying makeup on a photo. In this paper we introduce an augmented reality system that allows users to apply virtual makeup directly on their face using a physical applicator, simulating a virtual mirror experience. Facial features are detected and tracked using an RGBD camera and mapped to a normalized 2D facial mesh composed of 124 triangles. Finger touches on the face are also detected on the RGBD video stream, and used to store the applied makeup texture representation on the corresponding 2D facial triangle. Rendering the face with the virtual makeup is performed by back-projecting the makeup stored on the facial mesh to the image captured by the camera. Our initial prototype demonstrates the feasibility of our technique, that detects touches very accurately (about 2.2 mm), and that achieves real-time interactive performance (about 15 fps) when tracking and rendering makeup using a regular PC and an Intel RealSense RGBD camera.

[1]  Antonis A. Argyros,et al.  Full DOF tracking of a hand interacting with an object by modeling occlusions and physical constraints , 2011, 2011 International Conference on Computer Vision.

[2]  Rui Caseiro,et al.  High-Speed Tracking with Kernelized Correlation Filters , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[4]  Manolis I. A. Lourakis,et al.  Evolutionary Quasi-Random Search for Hand Articulations Tracking , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Jovan Popovic,et al.  Real-time hand-tracking with a color glove , 2009, SIGGRAPH '09.

[6]  Antti Oulasvirta,et al.  Interactive Markerless Articulated Hand Motion Tracking Using RGB and Depth Data , 2013, 2013 IEEE International Conference on Computer Vision.

[7]  Michael S. Brown,et al.  Example-Based Cosmetic Transfer , 2007, 15th Pacific Conference on Computer Graphics and Applications (PG'07).

[8]  Akihiko Hanafusa,et al.  Makeup Support System for Visually Impaired Persons: Overview of System Functions , 2010, ICCHP.

[9]  Luc Van Gool,et al.  Tracking a hand manipulating an object , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[10]  Carlos Hitoshi Morimoto,et al.  Espelho virtual interativo para simulação de maquiagem , 2014, IHC.

[11]  G. Kortuem,et al.  Reflectance Spectroscopy: Principles, Methods, Applications , 1969 .

[12]  Fred Nicolls,et al.  Locating Facial Features with an Extended Active Shape Model , 2008, ECCV.

[13]  Abu Saleh Md. Mahfujur Rahman,et al.  Augmented Rendering of Makeup Features in a Smart Interactive Mirror System for Decision Support in Cosmetic Products Selection , 2010, 2010 IEEE/ACM 14th International Symposium on Distributed Simulation and Real Time Applications.

[14]  Diego Gutierrez,et al.  Screen-space perceptual rendering of human skin , 2009, TAP.

[15]  Soo-Mi Choi,et al.  Interactive Cosmetic Makeup of a 3D Point-Based Face Model , 2008, IEICE Trans. Inf. Syst..

[16]  Marcelo Knörich Zuffo,et al.  Point and go: Exploring 3D virtual environments , 2012, 2012 IEEE Symposium on 3D User Interfaces (3DUI).

[17]  Takeo Kanade,et al.  Visual Tracking of High DOF Articulated Structures: an Application to Human Hand Tracking , 1994, ECCV.

[18]  Andreas Aristidou,et al.  Motion capture with constrained inverse kinematics for real-time hand tracking , 2010, 2010 4th International Symposium on Communications, Control and Signal Processing (ISCCSP).

[19]  Wen-Chieh Lin,et al.  Physically-based cosmetic rendering , 2013, I3D '13.

[20]  Koji Tsukada,et al.  Smart makeup system: supporting makeup using lifelog sharing , 2011, UbiComp '11.

[21]  Itiro Siio,et al.  Smart Makeup Mirror: Computer-Augmented Mirror to Aid Makeup Application , 2009, HCI.

[22]  Paulo R. S. Mendonça,et al.  Model-based 3D tracking of an articulated hand , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[23]  C J Burstone,et al.  Lip posture and its significance in treatment planning. , 1967, American journal of orthodontics.

[24]  Gaurav Sharma,et al.  Adaptive Digital Makeup , 2009, ISVC.

[25]  Dong Guo,et al.  Digital face makeup by example , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Paolo Dario,et al.  A Survey of Glove-Based Systems and Their Applications , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[27]  Mircea Nicolescu,et al.  A Review on Vision-Based Full DOF Hand Motion Estimation , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[28]  Carlos Hitoshi Morimoto,et al.  5* Magic Wand: An RGBD Camera-Based 5 DoF User Interface for 3D Interaction , 2015, 2015 XVII Symposium on Virtual and Augmented Reality.

[29]  Feng Guo,et al.  Featured Wand for 3D Interaction , 2007, 2007 IEEE International Conference on Multimedia and Expo.

[30]  Björn Stenger,et al.  Model-based hand tracking using a hierarchical Bayesian filter , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Marina Weber.  Reflectance Spectroscopy Principles Methods Applications , 2016 .

[32]  Jae Woo Kim,et al.  Spectrum‐Based Color Reproduction Algorithm for Makeup Simulation of 3D Facial Avatar , 2013 .