Deformable Motion 3D Reconstruction by Union of Regularized Subspaces

This paper presents an approach to jointly retrieve camera pose, time-varying 3D shape, and automatic clustering based on motion primitives, from incomplete 2D trajectories in a monocular video. We introduce the concept of order-varying temporal regularization in order to exploit video data, that can be indistinctly applied to the 3D shape evolution as well as to the similarities between images. This results in a union of regularized subspaces which effectively encodes the 3D shape deformation. All parameters are learned via augmented Lagrange multipliers, in a unified and unsupervised manner that does not assume any training data at all. Experimental validation is reported on human motion from sparse to dense shapes, providing more robust and accurate solutions than state-of-the-art approaches in terms of 3D reconstruction, while also obtaining motion grouping results.

[1]  Francesc Moreno-Noguer,et al.  DUST: Dual Union of Spatio-Temporal Subspaces for Monocular Multiple Object 3D Reconstruction , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Lourdes Agapito,et al.  Modal Space: A Physics-Based Model for Sequential Estimation of Time-Varying Shape from Monocular Video , 2016, Journal of Mathematical Imaging and Vision.

[3]  Richard Szeliski,et al.  Building Rome in a day , 2009, ICCV.

[4]  Zhenyue Zhang,et al.  Low-Rank Matrix Approximation with Manifold Regularization , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Chong-Ho Choi,et al.  A Procrustean Markov Process for Non-rigid Structure Recovery , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Lourdes Agapito,et al.  Dense Variational Reconstruction of Non-rigid Surfaces from Monocular Video , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Didier Stricker,et al.  Dense Batch Non-Rigid Structure from Motion in a Second , 2017, 2017 IEEE Winter Conference on Applications of Computer Vision (WACV).

[8]  Alessio Del Bue,et al.  Factorization for non-rigid and articulated structure using metric projections , 2009, CVPR.

[9]  Yi Ma,et al.  The Augmented Lagrange Multiplier Method for Exact Recovery of Corrupted Low-Rank Matrices , 2010, Journal of structural biology.

[10]  Chong-Ho Choi,et al.  Procrustean Normal Distribution for Non-Rigid Structure from Motion , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Aaron Hertzmann,et al.  Nonrigid Structure-from-Motion: Estimating Shape and Motion with Hierarchical Priors , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Francesc Moreno-Noguer,et al.  A scalable, efficient, and accurate solution to non-rigid structure from motion , 2018, Comput. Vis. Image Underst..

[13]  Francesc Moreno-Noguer,et al.  Real-time 3D reconstruction of non-rigid shapes with a single moving camera , 2016, Comput. Vis. Image Underst..

[14]  Yaser Sheikh,et al.  3D Reconstruction of a Moving Point from a Series of 2D Projections , 2010, ECCV.

[15]  Adrien Bartoli,et al.  Coarse-to-fine low-rank structure-from-motion , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Edward Y. Chang,et al.  Parallel Spectral Clustering in Distributed Systems , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Francesc Moreno-Noguer,et al.  Image Collection Pop-up: 3D Reconstruction and Clustering of Rigid and Non-rigid Categories , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[18]  Constantine Caramanis,et al.  Robust Matrix Completion and Corrupted Columns , 2011, ICML.

[19]  G. Sapiro,et al.  A collaborative framework for 3D alignment and classification of heterogeneous subvolumes in cryo-electron tomography. , 2013, Journal of structural biology.

[20]  Francesc Moreno-Noguer,et al.  Combining Local-Physical and Global-Statistical Models for Sequential Deformable Shape from Motion , 2016, International Journal of Computer Vision.

[21]  Takeo Kanade,et al.  Nonrigid Structure from Motion in Trajectory Space , 2008, NIPS.

[22]  Aleix M. Martínez,et al.  Kernel non-rigid structure from motion , 2011, 2011 International Conference on Computer Vision.

[23]  J. M. M. Montiel,et al.  3D Reconstruction of Non-Rigid Surfaces in Real-Time Using Wedge Elements , 2012, ECCV Workshops.

[24]  Emmanuel J. Candès,et al.  Exact Matrix Completion via Convex Optimization , 2009, Found. Comput. Math..

[25]  Aleix M. Martínez,et al.  Computing Smooth Time Trajectories for Camera and Deformable Shape in Structure from Motion with Occlusion , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Emmanuel J. Candès,et al.  A Singular Value Thresholding Algorithm for Matrix Completion , 2008, SIAM J. Optim..

[27]  Simon Lucey,et al.  Complex Non-rigid Motion 3D Reconstruction by Union of Subspaces , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[28]  Constantine Caramanis,et al.  Robust Matrix Completion with Corrupted Columns , 2011, ArXiv.

[29]  Francesc Moreno-Noguer,et al.  Learning Shape, Motion and Elastic Models in Force Space , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[30]  Hongdong Li,et al.  A simple prior-free method for non-rigid structure-from-motion factorization , 2012, CVPR.