Multiview 3D warps

Image registration and 3D reconstruction are fundamental computer vision and medical imaging problems. They are particularly challenging when the input data are images of a deforming body obtained by a single moving camera. We propose a new modelling framework, the multiview 3D warps. Existing models are twofold: they estimate inter-image warps which are often inconsistent between the different images and do not model the underlying 3D structure, or reconstruct just a sparse set of points. In contrast, our multiview 3D warps combine the advantages of both; they have an explicit 3D component and a set of 3D deformations combined with projection to 2D. They thus capture the dense deforming body's time-varying shape and camera pose. The advantages over the classical solutions are numerous: thanks to our feature-based estimation method for the multiview 3D warps, one can not only augment the original images but also retarget or clone the observed body's 3D deformations by changing the pose. Experimental results on simulated and real data are reported, confirming the advantages of our framework over existing methods.

[1]  Alessio Del Bue,et al.  Non-rigid structure from motion using ranklet-based tracking and non-linear optimization , 2007, Image Vis. Comput..

[2]  Daniel Rueckert,et al.  Nonrigid registration using free-form deformations: application to breast MR images , 1999, IEEE Transactions on Medical Imaging.

[3]  Kiriakos N. Kutulakos,et al.  Non-rigid structure from locally-rigid motion , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[4]  Vincent Lepetit,et al.  Fast Non-Rigid Surface Detection, Registration and Realistic Augmentation , 2008, International Journal of Computer Vision.

[5]  Alessio Del Bue,et al.  Piecewise Quadratic Reconstruction of Non-Rigid Surfaces from Monocular Sequences , 2010, ECCV.

[6]  Jean Duchon,et al.  Interpolation des fonctions de deux variables suivant le principe de la flexion des plaques minces , 1976 .

[7]  Henning Biermann,et al.  Recovering non-rigid 3D shape from image streams , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[8]  Florent Brunet,et al.  Feature-Driven Direct Non-Rigid Image Registration , 2010, International Journal of Computer Vision.

[9]  Nassir Navab,et al.  NURBS Warps , 2009, BMVC.

[10]  Tom Vercauteren,et al.  Diffeomorphic demons: Efficient non-parametric image registration , 2009, NeuroImage.

[11]  Adrien Bartoli,et al.  Implicit Non-Rigid Structure-from-Motion with Priors , 2008, Journal of Mathematical Imaging and Vision.

[12]  Aaron Hertzmann,et al.  Nonrigid Structure-from-Motion: Estimating Shape and Motion with Hierarchical Priors , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  João Paulo Costeira,et al.  Estimating 3D shape from degenerate sequences with missing data , 2009, Comput. Vis. Image Underst..

[14]  Andrew W. Fitzgibbon,et al.  Damped Newton algorithms for matrix factorization with missing data , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[15]  Alessio Del Bue,et al.  Bilinear Factorization via Augmented Lagrange Multipliers , 2010, ECCV.

[16]  Adrien Bartoli,et al.  Generalized Thin-Plate Spline Warps , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Pascal Fua,et al.  Surface Deformation Models for Nonrigid 3D Shape Recovery , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Timothy F. Cootes,et al.  Computing Accurate Correspondences across Groups of Images , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Fred L. Bookstein,et al.  Principal Warps: Thin-Plate Splines and the Decomposition of Deformations , 1989, IEEE Trans. Pattern Anal. Mach. Intell..