Video to fully automatic 3D hair model

Imagine taking a selfie video with your mobile phone and getting as output a 3D model of your head (face and 3D hair strands) that can be later used in VR, AR, and any other domain. State of the art hair reconstruction methods allow either a single photo (thus compromising 3D quality) or multiple views, but they require manual user interaction (manual hair segmentation and capture of fixed camera views that span full 360°). In this paper, we describe a system that can completely automatically create a reconstruction from any video (even a selfie video), and we don't require specific views, since taking your -90°, 90°, and full back views is not feasible in a selfie capture. In the core of our system, in addition to the automatization components, hair strands are estimated and deformed in 3D (rather than 2D as in state of the art) thus enabling superior results. We provide qualitative, quantitative, and Mechanical Turk human studies that support the proposed system, and show results on a diverse variety of videos (8 different celebrity videos, 9 selfie mobile videos, spanning age, gender, hair length, type, and styling).

[1]  Tal Hassner,et al.  Regressing Robust and Discriminative 3D Morphable Models with a Very Deep Neural Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Dan Wang,et al.  A novel coarse-to-fine hair segmentation method , 2011, Face and Gesture 2011.

[3]  Ming-Hsuan Yang,et al.  Multi-objective convolutional learning for face labeling , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Thabo Beeler,et al.  High-quality single-shot capture of facial geometry , 2010, ACM Trans. Graph..

[5]  Steve Marschner,et al.  A Survey on Hair Modeling: Styling, Simulation, and Rendering , 2007, IEEE Transactions on Visualization and Computer Graphics.

[6]  Ira Kemelmacher-Shlizerman,et al.  Total Moving Face Reconstruction , 2014, ECCV.

[7]  Fernando De la Torre,et al.  Supervised Descent Method and Its Applications to Face Alignment , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Long Quan,et al.  Sketching hairstyles , 2007, SBIM '07.

[9]  Dieter Fox,et al.  DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Ira Kemelmacher-Shlizerman,et al.  Face Reconstruction from a Single Image using a Single Reference Face Shape , 2009 .

[11]  Matan Sela,et al.  3D Face Reconstruction by Learning from Synthetic Data , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[12]  Ira Kemelmacher-Shlizerman,et al.  Head Reconstruction from Internet Photos , 2016, ECCV.

[13]  Matan Sela,et al.  Learning Detailed Face Reconstruction from a Single Image , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Ira Kemelmacher-Shlizerman,et al.  Transfiguring portraits , 2016, ACM Trans. Graph..

[15]  Kun Zhou,et al.  High-quality hair modeling from a single portrait photo , 2015, ACM Trans. Graph..

[16]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Hao Li,et al.  Avatar digitization from a single image for real-time rendering , 2017, ACM Trans. Graph..

[18]  Andrew W. Fitzgibbon,et al.  Real-time non-rigid reconstruction using an RGB-D camera , 2014, ACM Trans. Graph..

[19]  Andrew Jones,et al.  Digital Ira: creating a real-time photoreal digital actor , 2013, SIGGRAPH '13.

[20]  Mark Pauly,et al.  Dynamic 3D avatar creation from hand-held video input , 2015, ACM Trans. Graph..

[21]  Kun Zhou,et al.  Real-time facial animation with image-based dynamic avatars , 2016, ACM Trans. Graph..

[22]  Justus Thies,et al.  Real-time expression transfer for facial reenactment , 2015, ACM Trans. Graph..

[23]  Steve Marschner,et al.  Capturing hair assemblies fiber by fiber , 2009, ACM Trans. Graph..

[24]  Szymon Rusinkiewicz,et al.  Structure-aware hair capture , 2013, ACM Trans. Graph..

[25]  Li-Yi Wei,et al.  Capturing braided hairstyles , 2014, ACM Trans. Graph..

[26]  Paul Debevec,et al.  The Light Stages and Their Applications to Photoreal Digital Actors , 2012, SIGGRAPH 2012.

[27]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[28]  Chongyang Ma,et al.  Single-view hair modeling using a hairstyle database , 2015, ACM Trans. Graph..

[29]  Nuttapong Chentanez,et al.  3D hair model from small set of images , 2016, MIG.

[30]  Matthew Turk,et al.  A Morphable Model For The Synthesis Of 3D Faces , 1999, SIGGRAPH.

[31]  Sylvain Paris,et al.  Capture of hair geometry from multiple images , 2004, ACM Trans. Graph..

[32]  Hao Yang,et al.  A data-driven approach to four-view image-based hair modeling , 2017, ACM Trans. Graph..

[33]  Michael M. Kazhdan,et al.  Screened poisson surface reconstruction , 2013, TOGS.

[34]  Xiaogang Wang,et al.  Hierarchical face parsing via deep learning , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[35]  Chongyang Ma,et al.  Robust hair capture using simulated examples , 2014, ACM Trans. Graph..

[36]  Kun Zhou,et al.  Single-view hair modeling for portrait manipulation , 2012, ACM Trans. Graph..

[37]  Ieee Xplore,et al.  IEEE Transactions on Pattern Analysis and Machine Intelligence Information for Authors , 2022, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[38]  Michael Goesele,et al.  Multi-View Stereo for Community Photo Collections , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[39]  Ira Kemelmacher-Shlizerman,et al.  What Makes Tom Hanks Look Like Tom Hanks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[40]  A. Laurentini,et al.  The Visual Hull Concept for Silhouette-Based Image Understanding , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[41]  Frédo Durand,et al.  Hair photobooth: geometric and photometric acquisition of real hairstyles , 2008, ACM Trans. Graph..

[42]  Kun Zhou,et al.  Hair Interpolation for Portrait Morphing , 2013, Comput. Graph. Forum.

[43]  Zoran Popovic,et al.  The space of human body shapes: reconstruction and parameterization from range scans , 2003, ACM Trans. Graph..

[44]  Ira Kemelmacher-Shlizerman,et al.  Face reconstruction in the wild , 2011, 2011 International Conference on Computer Vision.

[45]  Yiying Tong,et al.  FaceWarehouse: A 3D Facial Expression Database for Visual Computing , 2014, IEEE Transactions on Visualization and Computer Graphics.

[46]  Vibhav Vineet,et al.  Conditional Random Fields as Recurrent Neural Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[47]  Hao Li,et al.  Photorealistic Facial Texture Inference Using Deep Neural Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Kun Zhou,et al.  Dynamic hair manipulation in images and videos , 2013, ACM Trans. Graph..

[49]  Larry S. Davis,et al.  Detection and analysis of hair , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[50]  Christian Rössl,et al.  Laplacian surface editing , 2004, SGP '04.

[51]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[52]  Kun Zhou,et al.  AutoHair: fully automatic hair modeling from a single image , 2016, ACM Trans. Graph..

[53]  Jianping Shi,et al.  Face Parsing via Recurrent Propagation , 2017, BMVC.

[54]  Marc Pollefeys,et al.  Semantic 3D Reconstruction of Heads , 2016, ECCV.

[55]  Shu Liang,et al.  3D Face Hallucination from a Single Depth Frame , 2014, 2014 2nd International Conference on 3D Vision.