Real-time avatar animation with dynamic face texturing

In this paper, we present a system to capture and animate a highly realistic avatar model of a user in real-time. The animated human model consists of a rigged 3D mesh and a texture map. The system is based on KinectV2 input which captures the skeleton of the current pose of the subject in order to animate the human shape model. An additional high-resolution RGB camera is used to capture the face for updating the texture map on each frame. With this combination of image based rendering with computer graphics we achieve photo-realistic animations in real-time. Additionally, this approach is well suited for networked scenarios, because of the low per frame amount of data to animate the model, which consists of motion capture parameters and a video frame. With experimental results, we demonstrate the high degree of realism of the presented approach.

[1]  Hans-Peter Seidel,et al.  Video-based characters: creating new human performances from a multi-view video database , 2011, ACM Trans. Graph..

[2]  Anna Hilsmann,et al.  Example-based Body Model Optimization and Skinning , 2016, Eurographics.

[3]  Martin D. Levine,et al.  Registering Multiview Range Data to Create 3D Computer Objects , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[4]  Gérard G. Medioni,et al.  Object modeling by registration of multiple range images , 1991, Proceedings. 1991 IEEE International Conference on Robotics and Automation.

[5]  Hans-Peter Seidel,et al.  Free-viewpoint video of human actors , 2003, ACM Trans. Graph..

[6]  Mark Pauly,et al.  Dynamic 3D avatar creation from hand-held video input , 2015, ACM Trans. Graph..

[7]  Hans-Peter Seidel,et al.  Seeing People in Different Light — Joint Shape , Motion , and Reflectance Capture , 2007 .

[8]  Kun Zhou,et al.  3D shape regression for real-time facial animation , 2013, ACM Trans. Graph..

[9]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[10]  Peter Eisert,et al.  Geometry-assisted image-based rendering for facial analysis and synthesis , 2006, Signal Process. Image Commun..

[11]  P. Schönemann,et al.  A generalized solution of the orthogonal procrustes problem , 1966 .

[12]  James Lau,et al.  Playable universal capture , 2006, SIGGRAPH '06.

[13]  Christian Theobalt,et al.  Reconstructing detailed dynamic face geometry from monocular video , 2013, ACM Trans. Graph..

[14]  Justus Thies,et al.  Real-time expression transfer for facial reenactment , 2015, ACM Trans. Graph..

[15]  Peter Eisert,et al.  Articulated 3D model tracking with on-the-fly texturing , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[16]  BlaisGérard,et al.  Registering Multiview Range Data to Create 3D Computer Objects , 1995 .

[17]  Peter Eisert,et al.  Realistic retargeting of facial video , 2014, CVMP.

[18]  Anna Hilsmann,et al.  Temporally Consistent Wide Baseline Facial Performance Capture via Image Warping , 2015, VMV.

[19]  Sebastian Thrun,et al.  SCAPE: shape completion and animation of people , 2005, SIGGRAPH '05.

[20]  Peter Eisert,et al.  Model-based 3D-motion estimation with illumination compensation , 1997 .

[21]  Anna Hilsmann,et al.  Pose Space Image Based Rendering , 2013, Comput. Graph. Forum.