Gradient Flows for Optimizing Triangular Mesh-based Surfaces: Applications to 3D Reconstruction Problems Dealing with Visibility

This article tackles the problem of using variational methods for evolving 3D deformable surfaces. We give an overview of gradient descent flows when the shape is represented by a triangular mesh-based surface, and we detail the gradients of two generic energy functionals which embody a number of energies used in mesh processing and computer vision. In particular, we show how to rigorously account for visibility in the surface optimization process. We present different applications including 3D reconstruction from multiple views for which the visibility is fundamental. The gradient correctly takes into account the visibility changes that occur when a surface moves; this forces the contours generated by the reconstructed surface to match with the apparent contours in the input images.

[1]  Daniel Cremers,et al.  Continuous Global Optimization in Multiview 3D Reconstruction , 2007, EMMCVPR.

[2]  Daniel Cremers,et al.  Anisotropic Minimal Surfaces Integrating Photoconsistency and Normal Information for Multiview Stereo , 2010, ECCV.

[3]  Henrik Aanæs,et al.  A variational analysis of shape from specularities using sparse data , 2004, Proceedings. 2nd International Symposium on 3D Data Processing, Visualization and Transmission, 2004. 3DPVT 2004..

[4]  Daniel Cremers,et al.  Continuous Global Optimization in Multiview 3D Reconstruction , 2007, International Journal of Computer Vision.

[5]  Roberto Cipolla,et al.  Multiview Photometric Stereo , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Sang Uk Lee,et al.  Multiview normal field integration using level set methods , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Charles M. Elliott,et al.  Finite elements on evolving surfaces , 2007 .

[8]  Niels Chr. Overgaard,et al.  A Geometric Formulation of Gradient Descent for Variational Problems with Moving Surfaces , 2005, Scale-Space.

[9]  Jean-Philippe Pons,et al.  A Lagrangian Approach to Dynamic Interfaces through Kinetic Triangulation of the Ambient Space , 2007, Comput. Graph. Forum.

[10]  Marcus A. Magnor,et al.  Weighted Minimal Hypersurface Reconstruction , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Rajiv Gupta,et al.  Linear Pushbroom Cameras , 1994, ECCV.

[12]  Daniel Cremers,et al.  Continuous ratio optimization via convex relaxation with applications to multiview 3D reconstruction , 2009, CVPR.

[13]  Gregory G. Slabaugh,et al.  Active polyhedron: surface evolution theory applied to deformable meshes , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[14]  Michel Barlaud,et al.  Using the Shape Gradient for Active Contour Segmentation: from the Continuous to the Discrete Formulation , 2006, Journal of Mathematical Imaging and Vision.

[15]  Peter F. Sturm,et al.  Joint Estimation of Shape and Reflectance using Multiple Images with Known Illumination Conditions , 2009, International Journal of Computer Vision.

[16]  Olivier D. Faugeras,et al.  Variational principles, surface evolution, PDEs, level set methods, and the stereo problem , 1998, IEEE Trans. Image Process..

[17]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[18]  Jean-Philippe Pons,et al.  Towards high-resolution large-scale multi-view stereo , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Pau Gargallo,et al.  Minimizing the Reprojection Error in Surface Reconstruction from Images , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[20]  Pau Gargallo I. Piracés.  Contributions to the Bayesian Approach to Multi-View Stereo. (Contributions à l'approche bayésienne pour la stéréovision multi-vues) , 2008 .

[21]  Jean-Philippe Pons,et al.  Fast Level Set Multi-View Stereo on Graphics Hardware , 2006, Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06).

[22]  Martin Jägersand,et al.  Variational Shape and Reflectance Estimation Under Changing Light and Viewpoints , 2006, ECCV.

[23]  Vladimir Kolmogorov,et al.  What energy functions can be minimized via graph cuts? , 2002, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Olivier D. Faugeras,et al.  Multi-View Stereo Reconstruction and Scene Flow Estimation with a Global Image-Based Matching Score , 2007, International Journal of Computer Vision.

[25]  Jovan Popović,et al.  Dynamic shape capture using multi-view photometric stereo , 2009, SIGGRAPH 2009.

[26]  Daniel Cremers,et al.  3-D Reconstruction of Shaded Objects from Multiple Images Under Unknown Illumination , 2008, International Journal of Computer Vision.

[27]  J. Sethian,et al.  Fronts propagating with curvature-dependent speed: algorithms based on Hamilton-Jacobi formulations , 1988 .

[28]  Stefano Soatto,et al.  Tales of shape and radiance in multiview stereo , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[29]  Olivier D. Faugeras,et al.  Modelling dynamic scenes by registering multi-view image sequences , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[30]  Anders Heyden,et al.  Reconstructing Open Surfaces from Image Data , 2006, International Journal of Computer Vision.

[31]  Jean-Philippe Pons,et al.  Minimizing the Multi-view Stereo Reprojection Error for Triangular Surface Meshes , 2008, BMVC.

[32]  Francis Schmitt,et al.  Silhouette and stereo fusion for 3D object modeling , 2003, Fourth International Conference on 3-D Digital Imaging and Modeling, 2003. 3DIM 2003. Proceedings..

[33]  Stefano Soatto,et al.  Stereoscopic Segmentation , 2001, ICCV.

[34]  S. Osher,et al.  Variational problems and PDEs on implicit surfaces , 2001, Proceedings IEEE Workshop on Variational and Level Set Methods in Computer Vision.

[35]  Roberto Cipolla,et al.  Multiview Stereo via Volumetric Graph-Cuts and Occlusion Robust Photo-Consistency , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Stefano Soatto,et al.  Variational multiframe stereo in the presence of specular reflections , 2002, Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission.

[37]  Mark Meyer,et al.  Discrete Differential-Geometry Operators for Triangulated 2-Manifolds , 2002, VisMath.

[38]  Mark Meyer,et al.  Implicit fairing of irregular meshes using diffusion and curvature flow , 1999, SIGGRAPH.

[39]  Stefano Soatto,et al.  Multi-View Stereo Reconstruction of Dense Shape and Complex Appearance , 2005, International Journal of Computer Vision.

[40]  Ross T. Whitaker,et al.  A Level-Set Approach to 3D Reconstruction from Range Data , 1998, International Journal of Computer Vision.

[41]  Daniel Cremers,et al.  Integration of Multiview Stereo and Silhouettes Via Convex Functionals on Convex Domains , 2008, ECCV.

[42]  Jean-Philippe Pons,et al.  Generalized Surface Flows for Mesh Processing , 2007 .

[43]  Daniel Cremers,et al.  Continuous ratio optimization via convex relaxation with applications to multiview 3D reconstruction , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[44]  Radu Horaud,et al.  TransforMesh : A Topology-Adaptive Mesh-Based Approach to Surface Evolution , 2007, ACCV.

[45]  S. Osher,et al.  Algorithms Based on Hamilton-Jacobi Formulations , 1988 .

[46]  Francis Schmitt,et al.  Silhouette and stereo fusion for 3D object modeling , 2003, Fourth International Conference on 3-D Digital Imaging and Modeling, 2003. 3DIM 2003. Proceedings..

[47]  Marc Pollefeys,et al.  Multi-view reconstruction using photo-consistency and exact silhouette constraints: a maximum-flow formulation , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[48]  Hong Qin,et al.  Shape Reconstruction from 3D and 2D Data Using PDE-Based Deformable Surfaces , 2004, ECCV.

[49]  Jean-Philippe Pons,et al.  Generalized Gradients: Priors on Minimization Flows , 2007, International Journal of Computer Vision.

[50]  Tianli Yu,et al.  Shape and View Independent Reflectance Map from Multiple Views , 2004, ECCV.

[51]  Szymon Rusinkiewicz,et al.  Efficiently combining positions and normals for precise 3D geometry , 2005, ACM Trans. Graph..

[52]  Marcus A. Magnor,et al.  Weighted Minimal Hypersurfaces and Their Applications in Computer Vision , 2004, ECCV.