Fast generation of realistic virtual humans

In this paper we present a complete pipeline to create ready-to-animate virtual humans by fitting a template character to a point set obtained by scanning a real person using multi-view stereo reconstruction. Our virtual humans are built upon a holistic character model and feature a detailed skeleton, fingers, eyes, teeth, and a rich set of facial blendshapes. Furthermore, due to the careful selection of techniques and technology, our reconstructed humans are quite realistic in terms of both geometry and texture. Since we represent our models as single-layer triangle meshes and animate them through standard skeleton-based skinning and facial blendshapes, our characters can be used in standard VR engines out of the box. By optimizing for computation time and minimizing manual intervention, our reconstruction pipeline is capable of processing whole characters in less than ten minutes.

[1]  Jihun Yu,et al.  Unconstrained realtime facial performance capture , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Michael J. Black,et al.  Home 3D body scans from noisy image and range data , 2011, 2011 International Conference on Computer Vision.

[3]  Marc Erich Latoschik,et al.  FakeMi: a fake mirror system for avatar embodiment studies , 2016, VRST.

[4]  Thabo Beeler,et al.  High-quality single-shot capture of facial geometry , 2010, ACM Trans. Graph..

[5]  Mark Pauly,et al.  Dynamic 3D avatar creation from hand-held video input , 2015, ACM Trans. Graph..

[6]  Paul Debevec,et al.  The Digital Emily project: photoreal facial modeling and animation , 2009, SIGGRAPH '09.

[7]  Marc Erich Latoschik,et al.  SIAMC: a socially immersive avatar mediated communication platform , 2016, VRST.

[8]  Jovan Popovic,et al.  Deformation transfer for triangle meshes , 2004, ACM Trans. Graph..

[9]  Horst Bischof,et al.  Rapid Skin: Estimating the 3D Human Pose and Shape in Real-Time , 2012, 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission.

[10]  Evan Suma Rosenberg,et al.  Just‐in‐time, viable, 3‐D avatars from scans , 2017, Comput. Animat. Virtual Worlds.

[11]  Michael J. Black,et al.  ClothCap , 2017, ACM Trans. Graph..

[12]  Mario Botsch,et al.  Accurate Face Reconstruction through Anisotropic Fitting and Eye Correction , 2015, VMV.

[13]  Maria V. Sanchez-Vives,et al.  First Person Experience of Body Transfer in Virtual Reality , 2010, PloS one.

[14]  Ari Shapiro,et al.  Avatar reshaping and automatic rigging using a deformable model , 2015, MIG.

[15]  Tabitha C. Peck,et al.  Putting yourself in the skin of a black avatar reduces implicit racial bias , 2013, Consciousness and Cognition.

[16]  Andrea Tagliasacchi,et al.  Dynamic 2D/3D Registration , 2014, Eurographics.

[17]  Derek Bradley,et al.  An anatomically-constrained local deformation model for monocular face capture , 2016, ACM Trans. Graph..

[18]  Michael J. Black,et al.  FAUST: Dataset and Evaluation for 3D Mesh Registration , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Justus Thies,et al.  Face2Face: real-time face capture and reenactment of RGB videos , 2019, Commun. ACM.

[20]  Stefanos Zafeiriou,et al.  Robust Discriminative Response Map Fitting with Constrained Local Models , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Yiying Tong,et al.  FaceWarehouse: A 3D Facial Expression Database for Visual Computing , 2014, IEEE Transactions on Visualization and Computer Graphics.

[22]  Marc Erich Latoschik,et al.  The effect of avatar realism in immersive social virtual realities , 2017, VRST.

[23]  Bernt Schiele,et al.  Building statistical shape spaces for 3D human modeling , 2015, Pattern Recognit..

[24]  Christian Theobalt,et al.  Reconstruction of Personalized 3D Face Rigs from Monocular Video , 2016, ACM Trans. Graph..

[25]  Kun Zhou,et al.  Displaced dynamic expression regression for real-time facial tracking and animation , 2014, ACM Trans. Graph..

[26]  Mel Slater,et al.  Body ownership causes illusory self-attribution of speaking and influences subsequent real speaking , 2014, Proceedings of the National Academy of Sciences.

[27]  Michael J. Black,et al.  Breathing life into shape , 2014, ACM Trans. Graph..

[28]  Michael J. Black,et al.  Estimating human shape and pose from a single image , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[29]  Zoran Popovic,et al.  The space of human body shapes: reconstruction and parameterization from range scans , 2003, ACM Trans. Graph..

[30]  Tao Ju,et al.  Mean value coordinates for closed triangular meshes , 2005, ACM Trans. Graph..

[31]  Thabo Beeler,et al.  High-quality single-shot capture of facial geometry , 2010, SIGGRAPH 2010.

[32]  Michael J. Black,et al.  SMPL: A Skinned Multi-Person Linear Model , 2023 .

[33]  Gérard G. Medioni,et al.  Rapid avatar capture and simulation using commodity depth sensors , 2014, Comput. Animat. Virtual Worlds.

[34]  Berthold K. P. Horn,et al.  Closed-form solution of absolute orientation using unit quaternions , 1987 .

[35]  S. Buss Introduction to Inverse Kinematics with Jacobian Transpose , Pseudoinverse and Damped Least Squares methods , 2004 .

[36]  P. Ekman,et al.  Facial action coding system , 2019 .

[37]  Michael J. Black,et al.  Combined discriminative and generative articulated pose and non-rigid shape estimation , 2007, NIPS.

[38]  Shu Liang,et al.  3D Face Hallucination from a Single Depth Frame , 2014, 2014 2nd International Conference on 3D Vision.

[39]  이균하,et al.  조합방식의 CHARACTER GENERATOR에 관한 연구 , 1972 .

[40]  Sebastian Thrun,et al.  SCAPE: shape completion and animation of people , 2005, SIGGRAPH '05.

[41]  Ligang Liu,et al.  Scanning 3D Full Human Bodies Using Kinects , 2012, IEEE Transactions on Visualization and Computer Graphics.

[42]  Daniel Cremers,et al.  CopyMe3D: Scanning and Printing Persons in 3D , 2013, GCPR.

[43]  Xin Tong,et al.  Automatic acquisition of high-fidelity facial performances using monocular videos , 2014, ACM Trans. Graph..

[44]  Justus Thies,et al.  Real-time expression transfer for facial reenactment , 2015, ACM Trans. Graph..

[45]  Justus Thies,et al.  Demo of Face2Face: real-time face capture and reenactment of RGB videos , 2016, SIGGRAPH Emerging Technologies.

[46]  Paul E. Debevec,et al.  Multiview face capture using polarized spherical gradient illumination , 2011, ACM Trans. Graph..

[47]  Mar González-Franco,et al.  The contribution of real-time mirror reflections of motor actions on virtual body ownership in an immersive virtual environment , 2010, 2010 IEEE Virtual Reality Conference (VR).

[48]  Matthew Turk,et al.  A Morphable Model For The Synthesis Of 3D Faces , 1999, SIGGRAPH.

[49]  Jovan Popovic,et al.  Automatic rigging and animation of 3D characters , 2007, ACM Trans. Graph..

[50]  Michael J. Black,et al.  Coregistration: Simultaneous Alignment and Modeling of Articulated 3D Shape , 2012, ECCV.

[51]  Jonathan T. Barron,et al.  3D self-portraits , 2013, ACM Trans. Graph..

[52]  Yangang Wang,et al.  Online modeling for realtime facial animation , 2013, ACM Trans. Graph..

[53]  Aaron Hertzmann,et al.  Eurographics/ Acm Siggraph Symposium on Computer Animation (2006) Learning a Correlated Model of Identity and Pose-dependent Body Shape Variation for Real-time Synthesis , 2022 .

[54]  Marc Erich Latoschik,et al.  Anthropomorphism and Illusion of Virtual Body Ownership , 2015, ICAT-EGVE.

[55]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[56]  Michael J. Black,et al.  MoSh: motion and shape capture from sparse markers , 2014, ACM Trans. Graph..

[57]  Jochen Lang,et al.  Estimation of human body shape and posture under clothing , 2013, Comput. Vis. Image Underst..

[58]  Hans-Peter Seidel,et al.  Animating deformable objects using sparse spacetime constraints , 2014, ACM Trans. Graph..

[59]  Michael J. Black,et al.  Detailed Full-Body Reconstructions of Moving People from Monocular RGB-D Sequences , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[60]  Hao Li,et al.  Realtime performance-based facial animation , 2011, ACM Trans. Graph..

[61]  M. Pauly,et al.  Example-based facial rigging , 2010, ACM Trans. Graph..

[62]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[63]  Ken-ichi Anjyo,et al.  Practice and Theory of Blendshape Facial Models , 2014, Eurographics.

[64]  Martin Klaudiny,et al.  Rapid one-shot acquisition of dynamic VR avatars , 2017, 2017 IEEE Virtual Reality (VR).

[65]  Hans-Peter Seidel,et al.  A Statistical Model of Human Pose and Body Shape , 2009, Comput. Graph. Forum.

[66]  Mark Pauly,et al.  Dynamic 2D/3D registration for the Kinect , 2013, SIGGRAPH '13.