Sparse transfer for facial shape-from-shading

A sparse transfer model was proposed to fuse a set of source face shapes in a selective way in order to assist the shape reconstruction of target face.A non-Lambertian reflectance model was formulated to model the interaction between light and the surface of human face.Extensive experiments were conducted to illustrate that our method can improve the performance of face shape reconstruction, especially when only a small number of target images are available. We present an image-based 3D face shape reconstruction method which transfers shape cues inferred from source face images to guide the reconstruction of the target face. Specifically, a sparse face shape adaption mechanism is used to generate a target-specific reference shape by adaptively and selectively combining source face shapes. This reference shape can also facilitate the reconstruction optimization for the target shape. As an off-line process, each source shape has been derived from a set of given sufficient source images (more than 9) based on a non-Lambertian reflectance model. Such a process allows for the existence of cast shadow and specularity, and more accurately infers the source shape. Guided by the target-specific reference shape, the shape of a target face can be estimated using a small number of images (even only one). The proposed reconstruction method refers to a lighting estimation and an albedo estimation for the target face. No standard 3D shape (such as the high-precision scanned 3D face) is required in the reconstruction process. Compared to the state-of-the-arts including the Photometric Stereo, Tensor Spline, the single reference based method, and the GEM algorithm, the proposed sparse transfer model can produce visually better facial details and obtain smaller reconstruction errors.

[1]  Ira Kemelmacher-Shlizerman,et al.  Face reconstruction in the wild , 2011, 2011 International Conference on Computer Vision.

[2]  Terence Sim,et al.  The CMU Pose, Illumination, and Expression (PIE) database , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[3]  David J. Kriegman,et al.  Acquiring linear subspaces for face recognition under variable lighting , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Jian-Huang Lai,et al.  Non-ideal class non-point light source quotient image for face relighting , 2011, Signal Process..

[5]  Rama Chellappa,et al.  Appearance Characterization of Linear Lambertian Objects, Generalized Photometric Stereo, and Illumination-Invariant Face Recognition , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Hanspeter Pfister,et al.  Face transfer with multilinear models , 2005, SIGGRAPH 2005.

[7]  Akihiro Sugimoto,et al.  Compact and Accurate 3-D Face Modeling Using an RGB-D Camera: Let's Open the Door to 3-D Video Conference , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[8]  Ira Kemelmacher-Shlizerman,et al.  Molding Face Shapes by Example , 2006, ECCV.

[9]  Rajat Raina,et al.  Efficient sparse coding algorithms , 2006, NIPS.

[10]  Melvyn L. Smith,et al.  3D face reconstructions from photometric stereo using near infrared and visible light , 2010, Comput. Vis. Image Underst..

[11]  Ping-Sing Tsai,et al.  Shape from Shading: A Survey , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  Minsik Lee,et al.  A robust real-time algorithm for facial shape recovery from a single image containing cast shadow under general, unknown lighting , 2013, Pattern Recognit..

[13]  David J. Kriegman,et al.  From Few to Many: Illumination Cone Models for Face Recognition under Variable Lighting and Pose , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  Fei Yang,et al.  Expression flow for 3D-aware face component transfer , 2011, SIGGRAPH 2011.

[15]  Ronen Basri,et al.  Lambertian Reflectance and Linear Subspaces , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[16]  Y. Censor,et al.  Parallel Optimization: Theory, Algorithms, and Applications , 1997 .

[17]  Steven M. Seitz,et al.  Shape and spatially-varying BRDFs from photometric stereo , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[18]  Ran He,et al.  Face shape recovery from a single image using CCA mapping between tensor spaces , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  In-So Kweon,et al.  Exploiting Shading Cues in Kinect IR Images for Geometry Refinement , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  Amnon Shashua,et al.  The Quotient Image: Class-Based Re-Rendering and Recognition with Varying Illuminations , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Alan Edelman,et al.  The Geometry of Algorithms with Orthogonality Constraints , 1998, SIAM J. Matrix Anal. Appl..

[22]  Thomas Vetter,et al.  Face Recognition Based on Fitting a 3D Morphable Model , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  Ira Kemelmacher-Shlizerman,et al.  Ieee Transactions on Pattern Analysis and Machine Intelligence 1 3d Face Reconstruction from a Single Image Using a Single Reference Face Shape , 2022 .

[24]  Edwin R. Hancock,et al.  Recovering Facial Shape Using a Statistical Model of Surface Normal Direction , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Stephen Lin,et al.  Shading-Based Shape Refinement of RGB-D Images , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Marwan Mattar,et al.  Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments , 2008 .

[27]  Thomas Vetter,et al.  A morphable model for the synthesis of 3D faces , 1999, SIGGRAPH.

[28]  Thabo Beeler,et al.  Real-time high-fidelity facial performance capture , 2015, ACM Trans. Graph..

[29]  Ashutosh Saxena,et al.  Make3D: Learning 3D Scene Structure from a Single Still Image , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Andrew W. Fitzgibbon,et al.  Real-time non-rigid reconstruction using an RGB-D camera , 2014, ACM Trans. Graph..

[31]  Jian-Huang Lai,et al.  Extraction of illumination invariant facial features from a single image using nonsubsampled contourlet transform , 2010, Pattern Recognit..

[32]  Peter Kovesi,et al.  Shapelets correlated with surface normals produce surfaces , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[33]  Raghu Machiraju,et al.  A bilinear illumination model for robust face recognition , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[34]  Xiaoou Tang,et al.  Facial Landmark Detection by Deep Multi-task Learning , 2014, ECCV.

[35]  Chong-Ho Choi,et al.  Real-time facial shape recovery from a single image under general, unknown lighting by rank relaxation , 2014, Comput. Vis. Image Underst..

[36]  Michael Lindenbaum,et al.  Shape Reconstruction of 3D Bilaterally Symmetric Surfaces , 2000, International Journal of Computer Vision.

[37]  Marios Savvides,et al.  Unconstrained Pose-Invariant Face Recognition Using 3D Generic Elastic Models , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[38]  Robert J. Woodham,et al.  Photometric Stereo: A Reflectance Map Technique For Determining Surface Orientation From Image Intensity , 1979, Optics & Photonics.

[39]  Rama Chellappa,et al.  Symmetric Shape-from-Shading Using Self-ratio Image , 2001, International Journal of Computer Vision.

[40]  Alan C. Bovik,et al.  Texas 3D Face Recognition Database , 2010, 2010 IEEE Southwest Symposium on Image Analysis & Interpretation (SSIAI).

[41]  Rama Chellappa,et al.  A Method for Enforcing Integrability in Shape from Shading Algorithms , 1988, IEEE Trans. Pattern Anal. Mach. Intell..

[42]  Liming Chen,et al.  A novel geometric facial representation based on multi-scale extended local binary patterns , 2011, Face and Gesture 2011.

[43]  Wotao Yin,et al.  A feasible method for optimization with orthogonality constraints , 2013, Math. Program..

[44]  Baba C. Vemuri,et al.  Non-Lambertian Reflectance Modeling and Shape Recovery of Faces Using Tensor Splines , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[45]  Jihun Yu,et al.  Realtime facial animation with on-the-fly correctives , 2013, ACM Trans. Graph..

[46]  Marios Savvides,et al.  Gender and Ethnicity Specific Generic Elastic Models from a Single 2D Image for Novel 2D Pose Face Synthesis and Recognition , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  R. Basri,et al.  Statistical Symmetric Shape from Shading for 3D Structure Recovery of Faces , 2004, eccv 2004.

[48]  Tal Hassner,et al.  Viewing Real-World Faces in 3D , 2013, 2013 IEEE International Conference on Computer Vision.