Real-time mimicking of estonian speaker's mouth movements on a 3D avatar using Kinect 2

In this paper an algorithm for the real-time imitation of lips movements of an Estonian speaker on a 3D avatar is proposed. This research is focused on obtaining the RGB and depth information from a Kinect 2 camera, processing it and moving the avatar accordingly. For the more stable extraction of information about lip shape and movement, cosine function is used. The depth info helps to determine the closest speaker and stabilize the movement of avatar and tracking of lips.

[1]  Angelos Barmpoutis,et al.  Tensor Body: Real-Time Reconstruction of the Human Body and Avatar Synthesis From RGB-D , 2013, IEEE Transactions on Cybernetics.

[2]  Rong Xiong,et al.  Facial feature points detecting based on Gaussian Mixture Models , 2015, Pattern Recognit. Lett..

[3]  Pallavi Khatri,et al.  Facial Expression Detection Techniques: Based on Viola and Jones Algorithm and Principal Component Analysis , 2015, 2015 Fifth International Conference on Advanced Computing & Communication Technologies.

[4]  Luc Van Gool,et al.  Real-time facial feature detection using conditional regression forests , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Jing Xu,et al.  Lip Detection and Tracking Using Variance Based Haar-Like Features and Kalman filter , 2010, 2010 Fifth International Conference on Frontier of Computer Science and Technology.

[7]  William McIlhagga,et al.  The Canny Edge Detector Revisited , 2011, International Journal of Computer Vision.

[8]  Shai Avidan,et al.  Extended Lucas-Kanade Tracking , 2014, ECCV.

[9]  Keiichi Abe,et al.  Topological structural analysis of digitized binary images by border following , 1985, Comput. Vis. Graph. Image Process..

[10]  S. Y. Chen,et al.  Kalman Filter for Robot Vision: A Survey , 2012, IEEE Transactions on Industrial Electronics.

[11]  Qiang Ji,et al.  Learning the Deep Features for Eye Detection in Uncontrolled Conditions , 2014, 2014 22nd International Conference on Pattern Recognition.

[12]  Alexandre Paz Mena,et al.  Comparative Study of the Features Used by Algorithms Based on Viola and Jones Face Detection Algorithm , 2015, IWINAC.

[13]  Thomas S. Huang,et al.  Interactive Facial Feature Localization , 2012, ECCV.

[14]  Alice Caplier.  Lip detection and tracking , 2001, Proceedings 11th International Conference on Image Analysis and Processing.

[15]  Sheng Yang,et al.  A fast mouth detection algorithm based on face organs , 2009, 2009 2nd International Conference on Power Electronics and Intelligent Transportation System (PEITS).