Learning 3D shape from a single facial image via non-linear manifold embedding and alignment

The 3D reconstruction of a face from a single frontal image is an ill-posed problem. This is further accentuated when the face image is captured under different poses and/or complex illumination conditions. In this paper, we aim to solve the shape recovery problem from a single facial image under these challenging conditions. The local image models for each patch of facial images and the local surface models for each patch of 3D shape are learned using a non-linear dimensionality reduction technique, and the correspondences between these local models are then learned by a manifold alignment method. By combining the local shapes, the global shape of a face can be reconstructed directly using a single least-square system of equations. We perform experiments on synthetic and real data, and validate the algorithm against the ground truth. Experimental results show that our method can yield accurate shape recovery from out-of-training samples with a variety of pose and illumination variations.

[1]  Berthold K. P. Horn,et al.  Shape from shading , 1989 .

[2]  Paul A. Griffin,et al.  Statistical Approach to Shape from Shading: Reconstruction of Three-Dimensional Face Surfaces from Single Two-Dimensional Images , 1996, Neural Computation.

[3]  J. Atick,et al.  Statistical Approach to Shape from Shading : Reconstruction of 3 D Face Surfaces from Single 2 D , 1997 .

[4]  Matthew Turk,et al.  A Morphable Model For The Synthesis Of 3D Faces , 1999, SIGGRAPH.

[5]  Amnon Shashua,et al.  The quotient image: Class based recognition and synthesis under varying illumination conditions , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[6]  Ping-Sing Tsai,et al.  Shape from Shading: A Survey , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  David J. Kriegman,et al.  From few to many: generative models for recognition under variable pose and illumination , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[8]  T. Kanade,et al.  Combining Models and Exemplars for Face Recognition: An Illuminating Example , 2001 .

[9]  Terence Sim,et al.  The CMU Pose, Illumination, and Expression (PIE) database , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[10]  Neil D. Lawrence,et al.  Gaussian Process Latent Variable Models for Visualisation of High Dimensional Data , 2003, NIPS.

[11]  Rama Chellappa,et al.  Symmetric Shape-from-Shading Using Self-ratio Image , 2001, International Journal of Computer Vision.

[12]  R. Basri,et al.  Statistical Symmetric Shape from Shading for 3D Structure Recovery of Faces , 2004, eccv 2004.

[13]  Jovan Popovic,et al.  Deformation transfer for triangle meshes , 2004, ACM Trans. Graph..

[14]  Michael Lindenbaum,et al.  Shape Reconstruction of 3D Bilaterally Symmetric Surfaces , 2000, International Journal of Computer Vision.

[15]  Sami Romdhani,et al.  Estimating 3D shape and texture using pixel intensity, edges, specular highlights, texture constraints and a prior , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[16]  Edwin R. Hancock,et al.  Recovering facial shape and albedo using a statistical model of surface normal direction , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[17]  Joe Y. Chang,et al.  Validation of an accelerated ‘demons’ algorithm for deformable image registration in radiation therapy , 2005, Physics in medicine and biology.

[18]  Olivier D. Faugeras,et al.  Shape from shading: a well-posed problem? , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[19]  Takeo Kanade,et al.  3D Alignment of Face in a Single Image , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[20]  Olivier D. Faugeras,et al.  A Unifying and Rigorous Shape from Shading Method Adapted to Realistic Data and Applications , 2006, Journal of Mathematical Imaging and Vision.

[21]  Edwin R. Hancock,et al.  A Simple Coupled Statistical Model for 3D Face Shape Recovery , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[22]  Ira Kemelmacher-Shlizerman,et al.  Molding Face Shapes by Example , 2006, ECCV.

[23]  Miguel Á. Carreira-Perpiñán,et al.  Non-rigid point set registration: Coherent Point Drift , 2006, NIPS.

[24]  Horst Bischof,et al.  3D and Infrared Face Reconstruction from RGB data using Canonical Correlation Analysis , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[25]  Olivier D. Faugeras,et al.  Shape From Shading , 2006, Handbook of Mathematical Models in Computer Vision.

[26]  Neil D. Lawrence,et al.  Learning for Larger Datasets with the Gaussian Process Latent Variable Model , 2007, AISTATS.

[27]  B. Schölkopf,et al.  Non-rigid point set registration: Coherent Point Drift , 2007 .

[28]  Ran He,et al.  Face shape recovery from a single image using CCA mapping between tensor spaces , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[29]  Pascal Fua,et al.  Local deformation models for monocular 3D shape recovery , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[30]  Sridhar Mahadevan,et al.  Manifold alignment using Procrustes analysis , 2008, ICML '08.

[31]  Iain Murray,et al.  Introduction to Gaussian Processes , 2008 .

[32]  Ruigang Yang,et al.  Illumination and Person-Insensitive Head Pose Estimation Using Distance Metric Learning , 2008, ECCV.