Detailed spatio-temporal reconstruction of eyelids

In recent years we have seen numerous improvements on 3D scanning and tracking of human faces, greatly advancing the creation of digital doubles for film and video games. However, despite the high-resolution quality of the reconstruction approaches available, current methods are unable to capture one of the most important regions of the face - the eye region. In this work we present the first method for detailed spatio-temporal reconstruction of eyelids. Tracking and reconstructing eyelids is extremely challenging, as this region exhibits very complex and unique skin deformation where skin is folded under while opening the eye. Furthermore, eyelids are often only partially visible and obstructed due to self-occlusion and eyelashes. Our approach is to combine a geometric deformation model with image data, leveraging multi-view stereo, optical flow, contour tracking and wrinkle detection from local skin appearance. Our deformation model serves as a prior that enables reconstruction of eyelids even under strong self-occlusions caused by rolling and folding skin as the eye opens and closes. The output is a person-specific, time-varying eyelid reconstruction with anatomically plausible deformations. Our high-resolution detailed eyelids couple naturally with current facial performance capture approaches. As a result, our method can largely increase the fidelity of facial capture and the creation of digital doubles.

[1]  Hans-Peter Seidel,et al.  Head shop: generating animated head models with anatomical structure , 2002, SCA '02.

[2]  Yu Zhang,et al.  Simulating Wrinkles in Facial Expressions on an Anatomy-Based Face , 2005, International Conference on Computational Science.

[3]  James F. O'Brien,et al.  Near-exhaustive precomputation of secondary cloth effects , 2013, ACM Trans. Graph..

[4]  Jun Li,et al.  Lightweight wrinkle synthesis for 3D facial modeling and animation , 2015, Comput. Aided Des..

[5]  Olga Sorkine-Hornung,et al.  On Linear Variational Surface Deformation Methods , 2008, IEEE Transactions on Visualization and Computer Graphics.

[6]  Thabo Beeler,et al.  High-quality single-shot capture of facial geometry , 2010, ACM Trans. Graph..

[7]  Kadi Bouatouch,et al.  Image-Based Modeling of the Human Eye , 2009, IEEE Transactions on Visualization and Computer Graphics.

[8]  Matthias Müller,et al.  Wrinkle meshes , 2010, SCA '10.

[9]  Thabo Beeler,et al.  Facial performance enhancement using dynamic shape space analysis , 2014, TOGS.

[10]  Byung-Uck Kim,et al.  A deformation transformer for real-time cloth animation , 2010, ACM Trans. Graph..

[11]  Hans-Peter Seidel,et al.  Shading-based dynamic shape refinement from multi-view video under general illumination , 2011, 2011 International Conference on Computer Vision.

[12]  John Hart,et al.  ACM Transactions on Graphics , 2004, SIGGRAPH 2004.

[13]  M. Otaduy,et al.  Multi-scale capture of facial geometry and motion , 2007, ACM Trans. Graph..

[14]  Pieter Peers,et al.  Rapid Acquisition of Specular and Diffuse Normal Maps from Polarized Spherical Gradient Illumination , 2007 .

[15]  Andrew Jones,et al.  Digital Ira: creating a real-time photoreal digital actor , 2013, SIGGRAPH '13.

[16]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[17]  Jörn Ostermann,et al.  Video-realistic image-based eye animation via statistically driven state machines , 2010, The Visual Computer.

[18]  Xin Tong,et al.  Automatic acquisition of high-fidelity facial performances using monocular videos , 2014, ACM Trans. Graph..

[19]  Iain Matthews,et al.  Modeling and animating eye blinks , 2011, TAP.

[20]  Ira Kemelmacher-Shlizerman,et al.  Total Moving Face Reconstruction , 2014, ECCV.

[21]  Garrison W. Cottrell,et al.  Transmitting and Decoding Facial Expressions , 2005, Psychological science.

[22]  Nadia Magnenat-Thalmann,et al.  A computational skin model: fold and wrinkle formation , 2002, IEEE Transactions on Information Technology in Biomedicine.

[23]  Kun Zhou,et al.  Displaced dynamic expression regression for real-time facial tracking and animation , 2014, ACM Trans. Graph..

[24]  GrossMarkus,et al.  Detailed spatio-temporal reconstruction of eyelids , 2015 .

[25]  Yangang Wang,et al.  Online modeling for realtime facial animation , 2013, ACM Trans. Graph..

[26]  Steve Maddock,et al.  Physically‐based forehead animation including wrinkles , 2015, Comput. Animat. Virtual Worlds.

[27]  Jinxiang Chai,et al.  Leveraging motion capture and 3D scanning for high-fidelity facial performance acquisition , 2011, SIGGRAPH 2011.

[28]  W. Heidrich,et al.  High resolution passive facial performance capture , 2010, ACM Trans. Graph..

[29]  Hans-Peter Seidel,et al.  Lightweight binocular facial performance capture under uncontrolled lighting , 2012, ACM Trans. Graph..

[30]  James F. O'Brien,et al.  Example-based wrinkle synthesis for clothing animation , 2010, ACM Trans. Graph..

[31]  Derek Bradley,et al.  Wrinkling Captured Garments Using Space‐Time Data‐Driven Deformation , 2009, Comput. Graph. Forum.

[32]  N. Badler,et al.  Eyes Alive Eyes Alive Eyes Alive Figure 1: Sample Images of an Animated Face with Eye Movements , 2022 .

[33]  Derek Bradley,et al.  High-quality passive facial performance capture using anchor frames , 2011, ACM Trans. Graph..

[34]  Peter J. Hunter,et al.  A virtual environment and model of the eye for surgical simulation , 1994, SIGGRAPH.

[35]  Thomas Brox,et al.  High Accuracy Optical Flow Estimation Based on a Theory for Warping , 2004, ECCV.

[36]  Paul A. Beardsley,et al.  Coupled 3D reconstruction of sparse facial hair and skin , 2012, ACM Trans. Graph..

[37]  Matthias Harders,et al.  Enriching coarse interactive elastic objects with high-resolution data-driven deformations , 2012, SCA '12.

[38]  Derek Bradley,et al.  High-quality capture of eyes , 2014, ACM Trans. Graph..

[39]  Taehyun Rhee,et al.  Real-time facial animation from live video tracking , 2011, SCA '11.

[40]  Jian Sun,et al.  Face Alignment by Explicit Shape Regression , 2012, International Journal of Computer Vision.

[41]  Matthew F. Peterson,et al.  Looking just below the eyes is optimal across face recognition tasks , 2012, Proceedings of the National Academy of Sciences.

[42]  Marc Alexa,et al.  As-rigid-as-possible surface modeling , 2007, Symposium on Geometry Processing.

[43]  Miguel A. Otaduy,et al.  Animating Wrinkles by Example on Non-Skinned Cloth , 2013, IEEE Transactions on Visualization and Computer Graphics.

[44]  Zhigang Deng,et al.  Live Speech Driven Head-and-Eye Motion Generators , 2012, IEEE Transactions on Visualization and Computer Graphics.

[45]  Kun Zhou,et al.  3D shape regression for real-time facial animation , 2013, ACM Trans. Graph..

[46]  Norman I. Badler,et al.  Look me in the Eyes: A Survey of Eye and Gaze Animation for Virtual Agents and Artificial Systems , 2014, Eurographics.

[47]  Saïda Bouakaz,et al.  Easy acquisition and real‐time animation of facial wrinkles , 2011, Comput. Animat. Virtual Worlds.

[48]  Markus H. Gross,et al.  Pose-space animation and transfer of facial details , 2008, SCA '08.

[49]  Takaaki Kuratate,et al.  A simple method for modeling wrinkles on human skin , 2002, 10th Pacific Conference on Computer Graphics and Applications, 2002. Proceedings..

[50]  Derek Bradley,et al.  Markerless garment capture , 2008, ACM Trans. Graph..

[51]  Jihun Yu,et al.  Realtime facial animation with on-the-fly correctives , 2013, ACM Trans. Graph..

[52]  Christian Theobalt,et al.  Reconstructing detailed dynamic face geometry from monocular video , 2013, ACM Trans. Graph..

[53]  Paul E. Debevec,et al.  Multiview face capture using polarized spherical gradient illumination , 2011, ACM Trans. Graph..

[54]  Martin Klaudiny,et al.  High-Detail 3D Capture and Non-sequential Alignment of Facial Performance , 2012, 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission.

[55]  Mark Pauly,et al.  Realtime performance-based facial animation , 2011, ACM Trans. Graph..

[56]  Marie-Paule Cani,et al.  Real-time dynamic wrinkles , 2004, Proceedings Computer Graphics International, 2004..

[57]  Jitendra Malik,et al.  Scale-Space and Edge Detection Using Anisotropic Diffusion , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[58]  John P. Lewis,et al.  Automated eye motion using texture synthesis , 2005, IEEE Computer Graphics and Applications.

[59]  Derek Bradley,et al.  Rigid stabilization of facial expressions , 2014, ACM Trans. Graph..

[60]  Alla Sheffer,et al.  Animation wrinkling: augmenting coarse cloth simulations with realistic-looking wrinkles , 2010, ACM Trans. Graph..

[61]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[62]  Pieter Peers,et al.  Facial performance synthesis using deformation-driven polynomial displacement maps , 2008, SIGGRAPH Asia '08.