Robust 3D face capture using example-based photometric stereo

We show that using example-based photometric stereo, it is possible to achieve realistic reconstructions of the human face. The method can handle non-Lambertian reflectance and attached shadows after a simple calibration step. We use spherical harmonics to model and de-noise the illumination functions from images of a reference object with known shape, and a fast grid technique to invert those functions and recover the surface normal for each point of the target object. The depth coordinate is obtained by weighted multi-scale integration of these normals, using an integration weight mask obtained automatically from the images themselves. We have applied these techniques to improve the PhotoFace system of Hansen et al. (2010).

[1]  Sang-Woong Lee,et al.  Noniterative 3D Face Reconstruction Based on Photometric Stereo , 2008, Int. J. Pattern Recognit. Artif. Intell..

[2]  Georgios Tziritas,et al.  Face Detection Using Quantized Skin Color Regions Merging and Wavelet Packet Analysis , 1999, IEEE Trans. Multim..

[3]  Steven M. Seitz,et al.  Example-based photometric stereo: shape reconstruction with general, varying BRDFs , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Jorge Stolfi,et al.  Matching Photometric Observation Vectors with Shadows and Variable Albedo , 2008, 2008 XXI Brazilian Symposium on Computer Graphics and Image Processing.

[5]  Azriel Rosenfeld,et al.  Face recognition: A literature survey , 2003, CSUR.

[6]  David J. Kriegman,et al.  ShadowCuts: Photometric Stereo with Shadows , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Paul E. Debevec,et al.  Single-shot photometric stereo by spectral multiplexing , 2011, ICCP.

[8]  Jorge Stolfi,et al.  A Uniform Grid Structure to Speed Up Example-Based Photometric Stereo , 2011, IEEE Transactions on Image Processing.

[9]  David J. Kriegman,et al.  From Few to Many: Illumination Cone Models for Face Recognition under Variable Lighting and Pose , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  Patrick J. Flynn,et al.  A survey of approaches and challenges in 3D and multi-modal 3D + 2D face recognition , 2006, Comput. Vis. Image Underst..

[11]  Thomas Vetter,et al.  A morphable model for the synthesis of 3D faces , 1999, SIGGRAPH.

[12]  Rama Chellappa,et al.  A Method for Enforcing Integrability in Shape from Shading Algorithms , 1988, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  James J. Little,et al.  Photometric stereo via locality sensitive high-dimension hashing , 2005, The 2nd Canadian Conference on Computer and Robot Vision (CRV'05).

[14]  Melvyn L. Smith,et al.  Face recognition in 2D and 2.5D using ridgelets and photometric stereo , 2012, Pattern Recognit..

[15]  Steven M. Seitz,et al.  Shape and spatially-varying BRDFs from photometric stereo , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[16]  Katsushi Ikeuchi,et al.  Consensus photometric stereo , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[17]  Jorge Stolfi,et al.  Multi-scale Integration of Slope Data on an Irregular Mesh , 2011, PSIVT.

[18]  David J. Kriegman,et al.  Photometric stereo with non-parametric and spatially-varying reflectance , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Xiaoou Tang,et al.  Robust 3D Face Recognition by Local Shape Difference Boosting , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Ira Kemelmacher-Shlizerman,et al.  Photometric Stereo with General, Unknown Lighting , 2006, International Journal of Computer Vision.

[21]  Steven M. Seitz,et al.  Shape and materials by example: a photometric stereo approach , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[22]  Alberto Del Bimbo,et al.  3D Face Recognition Using Isogeodesic Stripes , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  G McGunnigle.  Photometric stereo with gradated extended sources for recovery of specular surfaces. , 2010, Journal of the Optical Society of America. A, Optics, image science, and vision.

[24]  Piotr Indyk,et al.  Approximate nearest neighbors: towards removing the curse of dimensionality , 1998, STOC '98.

[25]  Rama Chellappa,et al.  Enforcing integrability by error correction using l1-minimization , 2009, CVPR.

[26]  R. Woodham.  Gradient and curvature from the photometric-stereo method, including local confidence estimation , 1994 .

[27]  Anil K. Jain,et al.  Face Detection in Color Images , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  Ronen Basri,et al.  Lambertian Reflectance and Linear Subspaces , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  Daniel Snow,et al.  Shape and albedo from multiple images using integrability , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[30]  Hideki Hayakawa.  Photometric stereo under a light source with arbitrary motion , 1994 .

[31]  Maria Petrou,et al.  The 4-Source Photometric Stereo Technique for Three-Dimensional Surfaces in the Presence of Highlights and Shadows , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  Vincent Nozick,et al.  Pyramidal Normal Map Integration for Real-time Photometric Stereo , 2010 .

[33]  Jorge Stolfi,et al.  Bases for Non-homogeneous Polynomial Ck Splines on the Sphere , 1998, LATIN.

[34]  Graham Fyffe,et al.  Single-shot photometric stereo by spectral multiplexing , 2010, 2011 IEEE International Conference on Computational Photography (ICCP).

[35]  R.J. Woodham,et al.  Determining surface curvature with photometric stereo , 1989, Proceedings, 1989 International Conference on Robotics and Automation.

[36]  Edwin R. Hancock,et al.  Facial Shape-from-shading and Recognition Using Principal Geodesic Analysis and Robust Statistics , 2007, International Journal of Computer Vision.

[37]  Sang Wook Lee,et al.  Photometric Stereo from Maximum Feasible Lambertian Reflections , 2010, ECCV.

[38]  Rainer Lienhart,et al.  An extended set of Haar-like features for rapid object detection , 2002, Proceedings. International Conference on Image Processing.

[39]  Edwin R. Hancock,et al.  Combinatorial Surface Integration , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[40]  Jorge Stolfi,et al.  A robust multi-scale integration method to obtain the depth from gradient maps , 2012, Comput. Vis. Image Underst..

[41]  Melvyn L. Smith,et al.  3D face reconstructions from photometric stereo using near infrared and visible light , 2010, Comput. Vis. Image Underst..

[42]  Rama Chellappa,et al.  What Is the Range of Surface Reconstructions from a Gradient Field? , 2006, ECCV.

[43]  Daniel Snow,et al.  Determining Generative Models of Objects Under Varying Illumination: Shape and Albedo from Multiple Images Using SVD and Integrability , 1999, International Journal of Computer Vision.

[44]  Melvyn L. Smith,et al.  2.5D Facial Expression Recognition using Photometric Stereo and the Area Weighted Histogram of Shape Index , 2012, 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication.

[45]  George Vogiatzis,et al.  Self-calibrated, Multi-spectral Photometric Stereo for 3D Face Capture , 2012, International Journal of Computer Vision.