Lightweight Photometric Stereo for Facial Details Recovery

Recently, 3D face reconstruction from a single image has achieved great success with the help of deep learning and shape prior knowledge, but they often fail to produce accurate geometry details. On the other hand, photometric stereo methods can recover reliable geometry details, but require dense inputs and need to solve a complex optimization problem. In this paper, we present a lightweight strategy that only requires sparse inputs or even a single image to recover high-fidelity face shapes with images captured under near-field lights. To this end, we construct a dataset containing 84 different subjects with 29 expressions under 3 different lights. Data augmentation is applied to enrich the data in terms of diversity in identity, lighting, expression, etc. With this constructed dataset, we propose a novel neural network specially designed for photometric stereo based 3D face reconstruction. Extensive experiments and comparisons demonstrate that our method can generate high-quality reconstruction results with one to three facial images captured under near-field lights. Our full framework is available at https://github.com/Juyong/FacePSNet.

[1]  Georgios Tzimiropoulos,et al.  Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[2]  Christian Wöhler,et al.  An introduction to image-based 3D surface reconstruction and a survey of photometric stereo methods , 2011 .

[3]  Syed Zulqarnain Gilani,et al.  Learning from Millions of 3 D Scans for Large-scale 3 D Face Recognition , 2018 .

[4]  Xi Zhou,et al.  Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network , 2018, ECCV.

[5]  Stefanos Zafeiriou,et al.  GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Andreas Lanitis,et al.  Image Based 3D Face Reconstruction: a Survey , 2009, Int. J. Image Graph..

[7]  Charlie C. L. Wang,et al.  Photometric stereo with near point lighting: A solution by mesh deformation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Chao Liu,et al.  Near-light photometric stereo using circularly placed point light sources , 2018, 2018 IEEE International Conference on Computational Photography (ICCP).

[9]  Xin Chen,et al.  Sparse Photometric 3D Face Reconstruction Guided by Morphable Models , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[10]  Yasuyuki Matsushita,et al.  Learning to Minify Photometric Stereo , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Daniel Cremers,et al.  LED-Based Photometric Stereo: Modeling, Calibration and Numerical Solution , 2017, Journal of Mathematical Imaging and Vision.

[12]  Matan Sela,et al.  Learning Detailed Face Reconstruction from a Single Image , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Wan-Chun Ma,et al.  The Digital Emily Project: Achieving a Photorealistic Digital Actor , 2010, IEEE Computer Graphics and Applications.

[14]  Jiaolong Yang,et al.  Accurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image to Image Set , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[15]  Mark Pauly,et al.  Dynamic 3D avatar creation from hand-held video input , 2015, ACM Trans. Graph..

[16]  Sami Romdhani,et al.  Optimal Step Nonrigid ICP Algorithms for Surface Registration , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Tal Hassner,et al.  Extreme 3D Face Reconstruction: Seeing Through Occlusions , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[18]  Pieter Peers,et al.  Rapid Acquisition of Specular and Diffuse Normal Maps from Polarized Spherical Gradient Illumination , 2007 .

[19]  King Ngi Ngan,et al.  MVF-Net: Multi-View 3D Face Morphable Model Regression , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[21]  Yasuyuki Matsushita,et al.  Deep Photometric Stereo Network , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[22]  Yongtian Wang,et al.  Robust Photometric Stereo via Low-Rank Matrix Completion and Recovery , 2010, ACCV.

[23]  Mingyuan Zhou,et al.  3D Face Reconstruction using Color Photometric Stereo with Uncalibrated Near Point Lights , 2019, 2020 IEEE International Conference on Computational Photography (ICCP).

[24]  Sami Romdhani,et al.  A 3D Face Model for Pose and Illumination Invariant Face Recognition , 2009, 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance.

[25]  Thomas Vetter,et al.  Face Recognition Based on Fitting a 3D Morphable Model , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[27]  Satoshi Ikehata,et al.  CNN-PS: CNN-based Photometric Stereo for General Non-Convex Surfaces , 2018, ECCV.

[28]  Jianfei Cai,et al.  Alive Caricature from 2D to 3D , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[29]  Robert J. Woodham,et al.  Photometric method for determining surface orientation from multiple images , 1980 .

[30]  Yasuyuki Matsushita,et al.  Self-Calibrating Deep Photometric Stereo Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Takanori Maehara,et al.  Neural Inverse Rendering for General Reflectance Photometric Stereo , 2018, ICML.

[32]  Bailin Deng,et al.  3D Face Reconstruction With Geometry Details From a Single Image , 2017, IEEE Transactions on Image Processing.

[33]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Xudong Jiang,et al.  SPLINE-Net: Sparse Photometric Stereo Through Lighting Interpolation and Normal Estimation Networks , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[35]  Kenny Mitchell,et al.  Photo-Realistic Facial Details Synthesis From Single Image , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[36]  Kai Han,et al.  PS-FCN: A Flexible Learning Framework for Photometric Stereo , 2018, ECCV.

[37]  Ioannis A. Kakadiaris,et al.  Multi-view 3D face reconstruction with deep recurrent neural networks , 2017, 2017 IEEE International Joint Conference on Biometrics (IJCB).

[38]  Kenny Mitchell,et al.  Feature-preserving detailed 3D face reconstruction from a single image , 2018, CVMP '18.

[39]  Ron Kimmel,et al.  Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[40]  Matthew Turk,et al.  A Morphable Model For The Synthesis Of 3D Faces , 1999, SIGGRAPH.

[41]  Justus Thies,et al.  Face2Face: real-time face capture and reenactment of RGB videos , 2019, Commun. ACM.

[42]  Yiying Tong,et al.  FaceWarehouse: A 3D Facial Expression Database for Visual Computing , 2014, IEEE Transactions on Visualization and Computer Graphics.

[43]  Paul E. Debevec,et al.  Effect of illumination on automatic expression recognition: A novel 3D relightable facial database , 2011, Face and Gesture 2011.

[44]  Syed Zulqarnain Gilani,et al.  Learning from Millions of 3D Scans for Large-Scale 3D Face Recognition , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[45]  Stefanos Zafeiriou,et al.  A 3D Morphable Model Learnt from 10,000 Faces , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Jianfei Cai,et al.  CNN-Based Real-Time Dense Face Reconstruction with Inverse-Rendered Photo-Realistic Face Images , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Michael J. Black,et al.  Learning to Regress 3D Face Shape and Expression From an Image Without 3D Supervision , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).