Simultaneous pose and non-rigid shape with particle dynamics

In this paper, we propose a sequential solution to simultaneously estimate camera pose and non-rigid 3D shape from a monocular video. In contrast to most existing approaches that rely on global representations of the shape, we model the object at a local level, as an ensemble of particles, each ruled by the linear equation of the Newton's second law of motion. This dynamic model is incorporated into a bundle adjustment framework, in combination with simple regularization components that ensure temporal and spatial consistency of the estimated shape and camera poses. The resulting approach is both efficient and robust to several artifacts such as noisy and missing data or sudden camera motions, while it does not require any training data at all. Validation is done in a variety of real video sequences, including articulated and non-rigid motion, both for continuous and discontinuous shapes. Our system is shown to perform comparable to competing batch, computationally expensive, methods and shows remarkable improvement with respect to the sequential ones.

[1]  Andrew J. Davison,et al.  Live dense reconstruction with a single moving camera , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[2]  Takeo Kanade,et al.  Shape and motion from image streams under orthography: a factorization method , 1992, International Journal of Computer Vision.

[3]  Adrien Bartoli,et al.  Sequential Non-Rigid Structure-from-Motion with the 3D-Implicit Low-Rank Shape Model , 2010, ECCV.

[4]  Dimitris N. Metaxas,et al.  Shape and Nonrigid Motion Estimation Through Physics-Based Synthesis , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Lourdes Agapito,et al.  Online Dense Non-Rigid 3D Shape and Camera Motion Recovery , 2014, BMVC.

[6]  Jing Xiao,et al.  A Closed-Form Solution to Non-Rigid Shape and Motion Recovery , 2004, International Journal of Computer Vision.

[7]  Pascal Fua,et al.  Template-free monocular reconstruction of deformable surfaces , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[8]  Aaron Hertzmann,et al.  Nonrigid Structure-from-Motion: Estimating Shape and Motion with Hierarchical Priors , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Chong-Ho Choi,et al.  Procrustean Normal Distribution for Non-Rigid Structure from Motion , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Richard Szeliski,et al.  Building Rome in a day , 2009, ICCV.

[11]  Michel Dhome,et al.  Generic and real-time structure from motion using local bundle adjustment , 2009, Image Vis. Comput..

[12]  Aleix M. Martínez,et al.  Non-rigid structure from motion with complementary rank-3 spaces , 2011, CVPR 2011.

[13]  Takeo Kanade,et al.  Nonrigid Structure from Motion in Trajectory Space , 2008, NIPS.

[14]  J. M. M. Montiel,et al.  Finite Element based sequential Bayesian Non-Rigid Structure from Motion , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Henning Biermann,et al.  Recovering non-rigid 3D shape from image streams , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[16]  Zoran Popovic,et al.  Physically based motion transformation , 1999, SIGGRAPH.

[17]  Raquel Urtasun,et al.  Physically-based motion models for 3D tracking: A convex formulation , 2011, 2011 International Conference on Computer Vision.

[18]  Lourdes Agapito,et al.  Factorization for non-rigid and articulated structure using metric projections , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Kiriakos N. Kutulakos,et al.  Non-rigid structure from locally-rigid motion , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[20]  Aleix M. Martínez,et al.  Kernel non-rigid structure from motion , 2011, 2011 International Conference on Computer Vision.

[21]  Odest Chadwicke Jenkins,et al.  Physical simulation for probabilistic motion tracking , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Sharat Chandran,et al.  Riemannian manifold optimisation for non-rigid structure from motion , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[23]  Lourdes Agapito,et al.  Dense Variational Reconstruction of Non-rigid Surfaces from Monocular Video , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  J. M. M. Montiel,et al.  3D Reconstruction of Non-Rigid Surfaces in Real-Time Using Wedge Elements , 2012, ECCV Workshops.

[25]  Hongdong Li,et al.  A Simple Prior-Free Method for Non-rigid Structure-from-Motion Factorization , 2012, International Journal of Computer Vision.

[26]  David J. Fleet,et al.  Estimating contact dynamics , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[27]  Francesc Moreno-Noguer,et al.  Probabilistic simultaneous pose and non-rigid shape recovery , 2011, CVPR 2011.

[28]  Lourdes Agapito,et al.  Good Vibrations: A Modal Analysis Approach for Sequential Non-rigid Structure from Motion , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[29]  David Baraff,et al.  Analytical methods for dynamic simulation of non-penetrating rigid bodies , 1989, SIGGRAPH.

[30]  S. Shankar Sastry,et al.  Optimization Criteria and Geometric Algorithms for Motion and Structure Estimation , 2001, International Journal of Computer Vision.

[31]  KanadeTakeo,et al.  Shape and motion from image streams under orthography , 1992 .

[32]  Yaser Sheikh,et al.  3D Reconstruction of a Moving Point from a Series of 2D Projections , 2010, ECCV.

[33]  Alessio Del Bue,et al.  Non-Rigid Metric Shape and Motion Recovery from Uncalibrated Images Using Priors , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[34]  J. Costeira,et al.  Optimal shape from motion estimation with missing and degenerate data , 2008, 2008 IEEE Workshop on Motion and video Computing.

[35]  Adrien Bartoli,et al.  Non-Rigid Shape-from-Motion for Isometric Surfaces using Infinitesimal Planarity , 2014, BMVC.

[36]  Lourdes Agapito,et al.  Energy based multiple model fitting for non-rigid structure from motion , 2011, CVPR 2011.

[37]  Adrien Bartoli,et al.  Coarse-to-fine low-rank structure-from-motion , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.