Adaptive 3D Face Reconstruction from Unconstrained Photo Collections

Given a collection of "in-the-wild" face images captured under a variety of unknown pose, expression, and illumination conditions, this paper presents a method for reconstructing a 3D face surface model of an individual along with albedo information. Motivated by the success of recent face reconstruction techniques on large photo collections, we extend prior work to adapt to low quality photo collections with fewer images. We achieve this by fitting a 3D Morphable Model to form a personalized template and developing a novel photometric stereo formulation, under a coarse-to-fine scheme. Superior experimental results are reported on synthetic and real-world photo collections.

[1]  Yiying Tong,et al.  Unconstrained 3D face reconstruction , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Stefanos Zafeiriou,et al.  Automatic construction Of robust spherical harmonic subspaces , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Roberto Cipolla,et al.  Multiview Photometric Stereo , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Ulrich Pinkall,et al.  Computing Discrete Minimal Surfaces and Their Conjugates , 1993, Exp. Math..

[5]  Richard Szeliski,et al.  Building Rome in a day , 2009, ICCV.

[6]  Xiaoming Liu,et al.  Attribute preserved face de-identification , 2015, 2015 International Conference on Biometrics (ICB).

[7]  Thomas Vetter,et al.  Face Recognition Based on Fitting a 3D Morphable Model , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Jürgen Beyerer,et al.  Adaptive Contour Fitting for Pose-Invariant 3D Face Shape Reconstruction , 2015, BMVC.

[9]  Junjie Yan,et al.  Learn to Combine Multiple Hypotheses for Accurate Face Alignment , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[10]  Yiying Tong,et al.  Adaptive 3D Face Reconstruction from Unconstrained Photo Collections , 2016, CVPR.

[11]  PaulyMark,et al.  Dynamic 3D avatar creation from hand-held video input , 2015 .

[12]  Christian Theobalt,et al.  Reconstructing detailed dynamic face geometry from monocular video , 2013, ACM Trans. Graph..

[13]  Derek Bradley,et al.  High-quality passive facial performance capture using anchor frames , 2011, ACM Trans. Graph..

[14]  Lei Zhang,et al.  Face recognition from a single training image under arbitrary unknown lighting using spherical harmonics , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Tsuhan Chen,et al.  Pose-robust face recognition using geometry assisted probabilistic modeling , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[16]  Mark Pauly,et al.  Dynamic 3D avatar creation from hand-held video input , 2015, ACM Trans. Graph..

[17]  Volker Blanz,et al.  Automated 3D Face Reconstruction from Multiple Images Using Quality Measures , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Takeo Kanade,et al.  Dense 3D face alignment from 2D videos in real-time , 2015, 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[19]  Stefanos Zafeiriou,et al.  300 Faces In-The-Wild Challenge: database and results , 2016, Image Vis. Comput..

[20]  FanHaoqiang,et al.  Approaching human level facial landmark localization by deep learning , 2016 .

[21]  Xiaoming Liu,et al.  Pose-Invariant 3D Face Alignment , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[22]  Dieter Fox,et al.  DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Ira Kemelmacher-Shlizerman,et al.  Photometric Stereo with General, Unknown Lighting , 2006, International Journal of Computer Vision.

[24]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[25]  Paul E. Debevec,et al.  Multiview face capture using polarized spherical gradient illumination , 2011, ACM Trans. Graph..

[26]  Zhiyong Feng,et al.  3D face modeling based on structure optimization and surface reconstruction with B-Spline , 2016, Neurocomputing.

[27]  Daniel Snow,et al.  Determining Generative Models of Objects Under Varying Illumination: Shape and Albedo from Multiple Images Using SVD and Integrability , 1999, International Journal of Computer Vision.

[28]  Carl D. Meyer,et al.  Ranking National Football League teams using Google's pagerank , 2006 .

[29]  Luc Van Gool,et al.  Face/Off: live facial puppetry , 2009, SCA '09.

[30]  Ira Kemelmacher-Shlizerman,et al.  Total Moving Face Reconstruction , 2014, ECCV.

[31]  Katsushi Ikeuchi,et al.  Photometric Stereo Using Internet Images , 2014, 2014 2nd International Conference on 3D Vision.

[32]  Ronen Basri,et al.  Accuracy of Spherical Harmonic Approximations for Images of Lambertian Objects under Far and Near Lighting , 2004, ECCV.

[33]  Ira Kemelmacher-Shlizerman,et al.  Internet Based Morphable Model , 2013, 2013 IEEE International Conference on Computer Vision.

[34]  Robert J. Woodham,et al.  Photometric method for determining surface orientation from multiple images , 1980 .

[35]  Andrew W. Fitzgibbon,et al.  3D scanning deformable objects with a single RGBD sensor , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Liming Chen,et al.  3D-Aided Face Recognition Robust to Expression and Pose Variations , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[37]  Kun Zhou,et al.  Displaced dynamic expression regression for real-time facial tracking and animation , 2014, ACM Trans. Graph..

[38]  Ira Kemelmacher-Shlizerman,et al.  Face reconstruction in the wild , 2011, 2011 International Conference on Computer Vision.

[39]  Christian Theobalt,et al.  Reconstruction of Personalized 3D Face Rigs from Monocular Video , 2016, ACM Trans. Graph..

[40]  Hideki Hayakawa.  Photometric stereo under a light source with arbitrary motion , 1994 .

[41]  Xin Tong,et al.  Automatic acquisition of high-fidelity facial performances using monocular videos , 2014, ACM Trans. Graph..

[42]  Alfred M. Bruckstein,et al.  Optimum fiducials under weak perspective projection , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[43]  PengWeilong,et al.  3D face modeling based on structure optimization and surface reconstruction with B-Spline , 2016 .

[44]  Lijun Yin,et al.  A high-resolution 3D dynamic facial expression database , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[45]  Yiying Tong,et al.  FaceWarehouse: A 3D Facial Expression Database for Visual Computing , 2014, IEEE Transactions on Visualization and Computer Graphics.

[46]  David J. Kriegman,et al.  Nine points of light: acquiring subspaces for face recognition under variable lighting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[47]  Chang Yang,et al.  Improving 3D Face Details Based on Normal Map of Hetero-source Images , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[48]  Justus Thies,et al.  Face2Face: Real-Time Face Capture and Reenactment of RGB Videos , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[50]  Xiaoming Liu,et al.  Large-Pose Face Alignment via CNN-Based Dense 3D Model Fitting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Jun Wang,et al.  3D Facial Expression Recognition Based on Primitive Surface Feature Distribution , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[52]  Thabo Beeler,et al.  High-quality single-shot capture of facial geometry , 2010, ACM Trans. Graph..

[53]  Ronen Basri,et al.  Lambertian Reflectance and Linear Subspaces , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[54]  Yongtian Wang,et al.  Robust Photometric Stereo via Low-Rank Matrix Completion and Recovery , 2010, ACCV.

[55]  Kun Zhou,et al.  Mesh editing with poisson-based gradient field manipulation , 2004, ACM Trans. Graph..

[56]  Sami Romdhani,et al.  A 3D Face Model for Pose and Illumination Invariant Face Recognition , 2009, 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance.

[57]  Thomas Vetter,et al.  A morphable model for the synthesis of 3D faces , 1999, SIGGRAPH.

[58]  Rajeev Motwani,et al.  The PageRank Citation Ranking : Bringing Order to the Web , 1999, WWW 1999.

[59]  Justus Thies,et al.  Real-time expression transfer for facial reenactment , 2015, ACM Trans. Graph..

[60]  Hans-Peter Seidel,et al.  Lightweight binocular facial performance capture under uncontrolled lighting , 2012, ACM Trans. Graph..

[61]  Andrew Jones,et al.  Driving High-Resolution Facial Scans with Video Performance Capture , 2014, ACM Trans. Graph..

[62]  Christian Rössl,et al.  Laplacian surface editing , 2004, SGP '04.

[63]  Justus Thies,et al.  Demo of Face2Face: real-time face capture and reenactment of RGB videos , 2016, SIGGRAPH Emerging Technologies.

[64]  Qingshan Liu,et al.  M3 CSR: Multi-view, multi-scale and multi-component cascade shape regression , 2016, Image Vis. Comput..

[65]  Melvyn L. Smith,et al.  3D face reconstructions from photometric stereo using near infrared and visible light , 2010, Comput. Vis. Image Underst..

[66]  Qijun Zhao,et al.  Examplar coherent 3D face reconstruction from forensic mugshot database , 2017, Image Vis. Comput..

[67]  Thabo Beeler,et al.  High-quality single-shot capture of facial geometry , 2010, SIGGRAPH 2010.

[68]  Kun Zhou,et al.  Mesh editing with poisson-based gradient field manipulation , 2004, SIGGRAPH 2004.

[69]  Marco Attene,et al.  ReMESH: An Interactive Environment to Edit and Repair Triangle Meshes , 2006, IEEE International Conference on Shape Modeling and Applications 2006 (SMI'06).

[70]  David J. Kriegman,et al.  Localizing parts of faces using a consensus of exemplars , 2011, CVPR.

[71]  Sébastien Marcel,et al.  Bob: a free signal processing and machine learning toolbox for researchers , 2012, ACM Multimedia.

[72]  Haoqiang Fan,et al.  Approaching human level facial landmark localization by deep learning , 2016, Image Vis. Comput..

[73]  Xiangyu Zhu,et al.  Face Alignment in Full Pose Range: A 3D Total Solution , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[74]  Thabo Beeler,et al.  Real-time high-fidelity facial performance capture , 2015, ACM Trans. Graph..

[75]  Xiangyu Zhu,et al.  High-fidelity Pose and Expression Normalization for face recognition in the wild , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[76]  Takeo Kanade,et al.  A Generative Shape Regularization Model for Robust Face Alignment , 2008, ECCV.