Template-free monocular reconstruction of deformable surfaces

It has recently been shown that deformable 3D surfaces could be recovered from single video streams. However, existing techniques either require a reference view in which the shape of the surface is known a priori, which often may not be available, or require tracking points over long sequences, which is hard to do. In this paper, we overcome these limitations. To this end, we establish correspondences between pairs of frames in which the shape is different and unknown. We then estimate homographies between corresponding local planar patches in both images. These yield approximate 3D reconstructions of points within each patch up to a scale factor. Since we consider overlapping patches, we can enforce them to be consistent over the whole surface. Finally, a local deformation model is used to fit a triangulated mesh to the 3D point cloud, which makes the reconstruction robust to both noise and outliers in the image data.

[1]  E. Malis,et al.  Deeper understanding of the homography decomposition for vision-based control , 2007 .

[2]  Alex Pentland,et al.  Closed-form solutions for physically-based shape modeling and recognition , 1991, Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[3]  Zhengyou Zhang,et al.  Iterative point matching for registration of free-form curves and surfaces , 1994, International Journal of Computer Vision.

[4]  Dmitry B. Goldgof,et al.  Nonrigid motion analysis based on dynamic refinement of finite element models , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[5]  Dimitris N. Metaxas,et al.  Constrained deformable superquadrics and nonrigid motion tracking , 1991, Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[6]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[7]  Jessica K. Hodgins,et al.  Estimating cloth simulation parameters from video , 2003, SCA '03.

[8]  René Vidal,et al.  Perspective Nonrigid Shape and Motion Recovery , 2008, ECCV.

[9]  Matthew Brand,et al.  Morphable 3D models from video , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[10]  A. Hanson,et al.  Scaled Euclidean 3D reconstruction based on externally uncalibrated cameras , 1995, Proceedings of International Symposium on Computer Vision - ISCV.

[11]  Alessio Del Bue,et al.  Non-rigid 3D Factorization for Projective Reconstruction , 2005, BMVC.

[12]  Thomas Vetter,et al.  A morphable model for the synthesis of 3D faces , 1999, SIGGRAPH.

[13]  Dimitris N. Metaxas,et al.  Shape and Nonrigid Motion Estimation Through Physics-Based Synthesis , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  Zenglin Xu,et al.  An Effective Approach to 3D Deformable Surface Tracking , 2008, ECCV.

[15]  Adrien Bartoli,et al.  Monocular Template-based Reconstruction of Inextensible Surfaces , 2011, International Journal of Computer Vision.

[16]  Nicholas Ayache,et al.  Frequency-Based Nonrigid Motion Analysis: Application to Four Dimensional Medical Images , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  Demetri Terzopoulos,et al.  A dynamic finite element surface model for segmentation and tracking in multidimensional medical images with application to cardiac 4D image analysis. , 1995, Computerized medical imaging and graphics : the official journal of the Computerized Medical Imaging Society.

[18]  Pascal Fua,et al.  Local deformation models for monocular 3D shape recovery , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Demetri Terzopoulos,et al.  Symmetry-seeking models and 3D object reconstruction , 1988, International Journal of Computer Vision.

[20]  M. Brand.  Morphable 3 D models from video , 2001 .

[21]  Dmitry B. Goldgof,et al.  Nonrigid Motion Analysis Based on Dynamic Refinement of Finite Element Models , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[22]  Aaron Hertzmann,et al.  Learning Non-Rigid 3D Shape from 2D Motion , 2003, NIPS.

[23]  Aaron Hertzmann,et al.  Nonrigid Structure-from-Motion: Estimating Shape and Motion with Hierarchical Priors , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[25]  Jing Xiao,et al.  Uncalibrated perspective reconstruction of deformable structures , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[26]  Adrien Bartoli,et al.  A Batch Algorithm for Implicit Non-rigid Shape and Motion Recovery , 2006, WDV.

[27]  Vincent Lepetit,et al.  Closed-Form Solution to Non-rigid 3D Surface Registration , 2008, ECCV.

[28]  Timothy F. Cootes,et al.  Active Appearance Models , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  Pascal Fua,et al.  Convex Optimization for Deformable Surface 3-D Tracking , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[30]  Jos F. Sturm,et al.  A Matlab toolbox for optimization over symmetric cones , 1999 .

[31]  Laurent D. Cohen,et al.  Finite-Element Methods for Active Contour Models and Balloons for 2-D and 3-D Images , 1993, IEEE Trans. Pattern Anal. Mach. Intell..