3D face modeling based on structure optimization and surface reconstruction with B-Spline

How to reconstruct 3D face model from wild photos is such a difficult issue that camera calibration is necessary and the images must be from video sequences. In this paper, a face reconstruction model with structure optimization is proposed to build 3D face surface with individual geometry and physical features reservation through wild face images directly and without camera calibration. Low rank and B-Spline are employed to estimate the aligned 2D structure, to calculate the depth information with SSIM, and to reconstruct the 3D face surface from control points and their space transformation. Furthermore, LFW and Bosphorus datasets, as well as Young-to-Aged samples, are introduced to verify the proposed approach and the experimental results demonstrate the feasibility and effectiveness even with different poses, expressions and age-variety. Highlights3D face structure optimization: frontal face structure optimization is considered as a sparse and low rank decomposition, and depth estimation is introduced as nonlinear programming based on constraints of multi-substructure.Face surface reconstruction is conducted with B-spline control grid deforming by 3D structure transformation.3D reconstruction solution is based on wild photos, instead of calibration.

[1]  Paul A. Beardsley,et al.  High-quality passive facial performance capture using anchor frames , 2011, SIGGRAPH 2011.

[2]  Ira Kemelmacher-Shlizerman,et al.  Face reconstruction in the wild , 2011, 2011 International Conference on Computer Vision.

[3]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[4]  Marios Savvides,et al.  In between 3D Active Appearance Models and 3D Morphable Models , 2009, 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[5]  Li Zhang,et al.  Spacetime faces: high resolution capture for modeling and animation , 2004, SIGGRAPH 2004.

[6]  Christian Theobalt,et al.  On-set performance capture of multiple actors with a stereo camera , 2013, ACM Trans. Graph..

[7]  Thabo Beeler,et al.  High-quality single-shot capture of facial geometry , 2010, SIGGRAPH 2010.

[8]  Takeo Kanade,et al.  Shape and motion from image streams under orthography: a factorization method , 1992, International Journal of Computer Vision.

[9]  Aleix M. Martínez,et al.  Rigid Structure from Motion from a Blind Source Separation Perspective , 2009, International Journal of Computer Vision.

[10]  Jörgen Ahlberg.  AN UPDATED PARAMETERISED FACE , 2001 .

[11]  Jean-Luc Dugelay,et al.  KinectFaceDB: A Kinect Database for Face Recognition , 2014, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[12]  Thomas Vetter,et al.  A morphable model for the synthesis of 3D faces , 1999, SIGGRAPH.

[13]  Lei Zhang,et al.  Fast Compressive Tracking , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Paul E. Debevec,et al.  Multiview face capture using polarized spherical gradient illumination , 2011, ACM Trans. Graph..

[15]  Huihui Song.  Robust visual tracking via online informative feature selection , 2014 .

[16]  Berthold K. P. Horn,et al.  Shape from shading , 1989 .

[17]  Ping-Sing Tsai,et al.  Shape from Shading: A Survey , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  Ira Kemelmacher-Shlizerman,et al.  Total Moving Face Reconstruction , 2014, ECCV.

[19]  Kin-Man Lam,et al.  Depth Estimation of Face Images Using the Nonlinear Least-Squares Model , 2013, IEEE Transactions on Image Processing.

[20]  Andrea Thelen,et al.  Improvements in Shape-From-Focus for Holographic Reconstructions With Regard to Focus Operators, Neighborhood-Size, and Height Value Interpolation , 2009, IEEE Transactions on Image Processing.

[21]  Rama Chellappa,et al.  Statistical bias in 3-D reconstruction from a monocular video , 2005, IEEE Transactions on Image Processing.

[22]  Luc Van Gool,et al.  Fast 3D Scanning with Automatic Motion Compensation , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[23]  T. Vetter,et al.  A statistical method for robust 3D surface reconstruction from sparse data , 2004 .

[24]  Jorge Nocedal,et al.  An Interior Point Algorithm for Large-Scale Nonlinear Programming , 1999, SIAM J. Optim..

[25]  Arman Savran,et al.  Bosphorus Database for 3D Face Analysis , 2008, BIOID.

[26]  Pietro Perona,et al.  Robust Face Landmark Estimation under Occlusion , 2013, 2013 IEEE International Conference on Computer Vision.

[27]  John Wright,et al.  RASL: Robust Alignment by Sparse and Low-Rank Decomposition for Linearly Correlated Images , 2012, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[29]  Zhiyong Feng,et al.  Robust visual tracking via online multiple instance learning with Fisher information , 2015, Pattern Recognit..

[30]  Olivier D. Faugeras,et al.  Shape from shading: a well-posed problem? , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[31]  Kin-Man Lam,et al.  Recovering the 3D shape and poses of face images based on the similarity transform , 2008, Pattern Recognit. Lett..

[32]  Richard Szeliski,et al.  Building Rome in a day , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[33]  Fernando De la Torre,et al.  Learning a generic 3D face model from 2D image databases using incremental Structure-from-Motion , 2010, Image Vis. Comput..

[34]  Steven M. Seitz,et al.  Photo tourism: exploring photo collections in 3D , 2006, ACM Trans. Graph..

[35]  Yiying Tong,et al.  Unconstrained 3D face reconstruction , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Ira Kemelmacher-Shlizerman,et al.  Ieee Transactions on Pattern Analysis and Machine Intelligence 1 3d Face Reconstruction from a Single Image Using a Single Reference Face Shape , 2022 .