Deformable 3D Reconstruction with an Object Database

Deformable 3D reconstruction from 2D images requires prior knowledge on the scene structure. Template-free methods use generic prior knowledge such as piecewise smoothness but require multiple images with significant baseline. Template-based methods require only one image but handle only one object for which they need specific prior knowledge, namely a 3D template. We here propose a novel method that alleviates the strong assumptions of both the template-free and template-based methods: our method uses multiple templates to achieve deformable 3D reconstruction from only one image and for multiple objects. It uses object recognition to automatically discover what objects are visible in the input image and to select the appropriate templates for deformable 3D reconstruction. The object database is built offline. Crucially, this database does not only contain appearance descriptors as in existing object recognition frameworks, but also material properties to facilitate deformable 3D reconstruction. We show successful experimental results with objects made of various materials such as paper, cloth and plastic.

[1]  Alessio Del Bue,et al.  Multiview 3D warps , 2011, 2011 International Conference on Computer Vision.

[2]  Javier Civera,et al.  Towards semantic SLAM using a monocular camera , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[3]  Michael M. Kazhdan,et al.  Poisson surface reconstruction , 2006, SGP '06.

[4]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[5]  Takeo Kanade,et al.  Trajectory Space: A Dual Representation for Nonrigid Structure from Motion , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Kiriakos N. Kutulakos,et al.  Non-rigid structure from locally-rigid motion , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[7]  Adrien Bartoli,et al.  Sequential Non-Rigid Structure-from-Motion with the 3D-Implicit Low-Rank Shape Model , 2010, ECCV.

[8]  Éric Marchand,et al.  Toward augmenting everything: Detecting and tracking geometrical features on planar objects , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[9]  J. M. M. Montiel,et al.  Finite Element based sequential Bayesian Non-Rigid Structure from Motion , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[11]  Adrien Bartoli,et al.  Maximizing the Predictivity of Smooth Deformable Image Warps through Cross-Validation , 2008, Journal of Mathematical Imaging and Vision.

[12]  Richard Szeliski,et al.  Towards Internet-scale multi-view stereo , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[13]  Aaron Hertzmann,et al.  Nonrigid Structure-from-Motion: Estimating Shape and Motion with Hierarchical Priors , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Adrien Bartoli,et al.  Generalized Thin-Plate Spline Warps , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Richard Szeliski,et al.  Modeling the World from Internet Photo Collections , 2008, International Journal of Computer Vision.

[16]  David G. Lowe,et al.  Fast Approximate Nearest Neighbors with Automatic Algorithm Configuration , 2009, VISAPP.

[17]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[18]  Adrien Bartoli,et al.  On template-based reconstruction from a single view: Analytical solutions and proofs of well-posedness for developable, isometric and conformal surfaces , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  A. Bartoli,et al.  Locally Affine and Planar Deformable Surface Reconstruction from Video , 2010 .

[20]  Lourdes Agapito,et al.  Energy based multiple model fitting for non-rigid structure from motion , 2011, CVPR 2011.

[21]  Richard Szeliski,et al.  Building Rome in a day , 2009, ICCV.

[22]  Hideo Saito,et al.  Virtually augmenting hundreds of real pictures: An approach based on learning, retrieval, and tracking , 2010, 2010 IEEE Virtual Reality Conference (VR).

[23]  Alessio Del Bue,et al.  Non-Rigid Metric Shape and Motion Recovery from Uncalibrated Images Using Priors , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[24]  Pascal Fua,et al.  Reconstructing sharply folding surfaces: A convex formulation , 2009, CVPR.

[25]  Daniel Pizarro-Perez,et al.  Feature-Based Deformable Surface Detection with Self-Occlusion Reasoning , 2011, International Journal of Computer Vision.

[26]  Adrien Bartoli,et al.  Monocular Template-based Reconstruction of Inextensible Surfaces , 2011, International Journal of Computer Vision.

[27]  Nassir Navab,et al.  Monocular Template-Based Reconstruction of Smooth and Inextensible Surfaces , 2010, ACCV.

[28]  David W. Murray,et al.  Keyframe-based recognition and localization during video-rate parallel tracking and mapping , 2011, Image Vis. Comput..

[29]  Bruno Lévy,et al.  Least squares conformal maps for automatic texture atlas generation , 2002, ACM Trans. Graph..

[30]  David Nistér,et al.  Scalable Recognition with a Vocabulary Tree , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[31]  Luc Van Gool,et al.  SURF: Speeded Up Robust Features , 2006, ECCV.

[32]  Henning Biermann,et al.  Recovering non-rigid 3D shape from image streams , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[33]  H. Chandler.  Database , 1985 .

[34]  Gabriel Taubin,et al.  The ball-pivoting algorithm for surface reconstruction , 1999, IEEE Transactions on Visualization and Computer Graphics.

[35]  Pascal Fua,et al.  Template-free monocular reconstruction of deformable surfaces , 2009, 2009 IEEE 12th International Conference on Computer Vision.