Fast 3D Reconstruction of Faces with Glasses

We present a method for the fast 3D face reconstruction of people wearing glasses. Our method explicitly and robustly models the case in which a face to be reconstructed is partially occluded by glasses. We propose a simple and generic model for glasses that copes with a wide variety of different shapes, colors and styles, without the need for any database or learning. Our algorithm is simple, fast and requires only small amounts of both memory and runtime resources, allowing for a fast interactive 3D reconstruction on commodity mobile phones. The thorough evaluation of our approach on synthetic and real data demonstrates superior reconstruction results due to the explicit modeling of glasses.

[1]  Thomas W. Reps,et al.  An Incremental Algorithm for a Generalization of the Shortest-Path Problem , 1996, J. Algorithms.

[2]  Thomas Brox,et al.  Global, Dense Multiscale Reconstruction for a Billion Points , 2015, ICCV.

[3]  Torsten Sattler,et al.  3D Modeling on the Go: Interactive 3D Reconstruction of Large-Scale Scenes on Mobile Devices , 2015, 2015 International Conference on 3D Vision.

[4]  Pushmeet Kohli,et al.  Real-Time Face Reconstruction from a Single Depth Image , 2014, 2014 2nd International Conference on 3D Vision.

[5]  Paolo Cignoni,et al.  Metro: Measuring Error on Simplified Surfaces , 1998, Comput. Graph. Forum.

[6]  Hanspeter Pfister,et al.  Face transfer with multilinear models , 2005, SIGGRAPH 2005.

[7]  Ira Kemelmacher-Shlizerman,et al.  Total Moving Face Reconstruction , 2014, ECCV.

[8]  Tony F. Chan,et al.  Mathematical Models for Local Nontexture Inpaintings , 2002, SIAM J. Appl. Math..

[9]  Bernhard Egger,et al.  Occlusion-aware 3D Morphable Face Models , 2016, BMVC.

[10]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[11]  Kostas Daniilidis,et al.  A Unifying Theory for Central Panoramic Systems and Practical Applications , 2000, ECCV.

[12]  Xavier P. Burgos-Artizzu,et al.  Real-time expression-sensitive HMD face reconstruction , 2015, SIGGRAPH Asia Technical Briefs.

[13]  Daniel Cremers,et al.  Tree Shape Priors with Connectivity Constraints Using Convex Relaxation on General Graphs , 2013, ICCV.

[14]  Patrick Rives,et al.  Single View Point Omnidirectional Camera Calibration from Planar Grids , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[15]  William J. Christmas,et al.  Fitting 3D Morphable Face Models using local features , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[16]  Yiying Tong,et al.  Adaptive 3D Face Reconstruction from Unconstrained Photo Collections , 2016, CVPR.

[17]  Marc Pollefeys,et al.  Face Reconstruction on Mobile Devices Using a Height Map Shape Model and Fast Regularization , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[18]  Xavier Bresson,et al.  Fast Global Minimization of the Active Contour/Snake Model , 2007, Journal of Mathematical Imaging and Vision.

[19]  M. Goesele,et al.  Floating scale surface reconstruction , 2014, ACM Trans. Graph..

[20]  Luc Van Gool,et al.  A Generalized EM Approach for 3D Model Based Face Recognition under Occlusions , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[21]  Marc Pollefeys,et al.  Semantic 3D Reconstruction of Heads , 2016, ECCV.

[22]  Daniel Cremers,et al.  Continuous Global Optimization in Multiview 3D Reconstruction , 2007, EMMCVPR.

[23]  Antonin Chambolle,et al.  Diagonal preconditioning for first order primal-dual algorithms in convex optimization , 2011, 2011 International Conference on Computer Vision.

[24]  Lujo Bauer,et al.  Accessorize to a Crime: Real and Stealthy Attacks on State-of-the-Art Face Recognition , 2016, CCS.

[25]  Andreas Geiger,et al.  Object scene flow for autonomous vehicles , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Thabo Beeler,et al.  Real-time high-fidelity facial performance capture , 2015, ACM Trans. Graph..

[27]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[28]  C. Zach.  Fast and High Quality Fusion of Depth Maps , 2008 .

[29]  Pushmeet Kohli,et al.  MobileFusion: Real-Time Volumetric Surface Reconstruction and Dense Tracking on Mobile Phones , 2015, IEEE Transactions on Visualization and Computer Graphics.

[30]  T. Cullen,et al.  Global existence of solutions for the relativistic Boltzmann equation on the flat Robertson-Walker space-time for arbitrarily large intial data , 2005, gr-qc/0507035.

[31]  Jean-Philippe Pons,et al.  Minimizing the Multi-view Stereo Reprojection Error for Triangular Surface Meshes , 2008, BMVC.

[32]  Radu Danescu,et al.  Eyeglasses Lens Contour Extraction from Facial Images Using an Efficient Shape Description , 2013, Sensors.

[33]  Jean-Philippe Pons,et al.  Efficient Multi-View Reconstruction of Large-Scale Scenes using Interest Points, Delaunay Triangulation and Graph Cuts , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[34]  Marc Levoy,et al.  A volumetric method for building complex models from range images , 1996, SIGGRAPH.

[35]  Thomas Vetter,et al.  Expression invariant 3D face recognition with a Morphable Model , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[36]  Victor S. Lempitsky,et al.  Global Optimization for Shape Fitting , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[37]  Jan-Michael Frahm,et al.  Structure-from-Motion Revisited , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Jean-Philippe Pons,et al.  Towards high-resolution large-scale multi-view stereo , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[39]  Marc Pollefeys,et al.  Turning Mobile Phones into 3D Scanners , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[40]  Sami Romdhani,et al.  A 3D Face Model for Pose and Illumination Invariant Face Recognition , 2009, 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance.

[41]  Horst Bischof,et al.  A Globally Optimal Algorithm for Robust TV-L1 Range Image Integration , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[42]  Marc Pollefeys,et al.  Live Metric 3D Reconstruction on Mobile Phones , 2013, 2013 IEEE International Conference on Computer Vision.

[43]  Daniel Cremers,et al.  Generalized Connectivity Constraints for Spatio-temporal 3D Reconstruction , 2014, ECCV.

[44]  Horst Bischof,et al.  Variational segmentation of elongated volumetric structures , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[45]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[46]  Marc Niethammer,et al.  Globally Optimal Finsler Active Contours , 2009, DAGM-Symposium.

[47]  Jian Sun,et al.  Face Alignment at 3000 FPS via Regressing Local Binary Features , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[48]  Helder Araújo,et al.  Issues on the geometry of central catadioptric image formation , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[49]  William A. P. Smith,et al.  3D morphable face models revisited , 2009, CVPR.

[50]  Simon Baker,et al.  Lucas-Kanade 20 Years On: A Unifying Framework , 2004, International Journal of Computer Vision.

[51]  Michael M. Kazhdan,et al.  Poisson surface reconstruction , 2006, SGP '06.

[52]  Thomas Brox,et al.  Global, Dense Multiscale Reconstruction for a Billion Points , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[53]  Peter Ondr.  MobileFusion: Real-time Volumetric Surface Reconstruction and Dense Tracking On Mobile Phones , 2015 .

[54]  Harry Shum,et al.  Automatic eyeglasses removal from face images , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[55]  Jean-Philippe Pons,et al.  Towards high-resolution large-scale multi-view stereo , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[56]  Sven Koenig,et al.  Incremental A* , 2001, NIPS.

[57]  Alan Brunton,et al.  Multilinear Wavelets: A Statistical Shape Space for Human Faces , 2014, ECCV.

[58]  Justus Thies,et al.  Demo of Face2Face: real-time face capture and reenactment of RGB videos , 2016, SIGGRAPH Emerging Technologies.

[59]  Rubén Usamentiaga,et al.  Glasses detection on real images based on robust alignment , 2015, Machine Vision and Applications.

[60]  Thomas Vetter,et al.  A morphable model for the synthesis of 3D faces , 1999, SIGGRAPH.

[61]  Oswald Aldrian,et al.  Inverse Rendering of Faces with a 3D Morphable Model , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.