NRSfPP: non-rigid structure-from-perspective projection

A state-of-the-art algorithm for perspective projection reconstruction of non-rigid surfaces from single-view and realistic videos is proposed. It overcomes the limitations arising from the usage of orthographic camera model and also the complexity and non-linearity issues of perspective projection equation. Unlike traditional non-rigid structure-from-motion (NRSfM) methods, which have been studied only on synthetic datasets and controlled lab environments that require some prior constraints (such as manually segmented objects, limited rotations and occlusions, and full-length trajectories); the proposed method can be used in realistic video sequences. In addition, contrary to previous methods that use multiple cameras with different relative viewing angles, only a single-view video is required to reconstruct the 3D structures. By only using the 2D frames of incoming video stream, the proposed method extracts the projective depth coefficients of each point in each input frame, rotation matrix, translation vector, varying camera parameters (such as focal lengths for each input frame), and finally reconstructs the 3D deformable shape. Due to the high number of unknowns, the problem has been divided into two parts of projective depth coefficients extraction and 3D shape reconstruction. Perspective reconstruction of non-rigid surfaces has been extended to be certainly converged which leads to a significant increase in execution frequency of the iterated algorithm that has been presented for projective depth coefficients extraction. As such, it produces promising results for perspective projection reconstruction of non-rigid surfaces from single-view and realistic videos. The accuracy and robustness of the proposed method is demonstrated quantitatively on synthetic data and qualitatively on real image sequences. The experimental results show that NRSfPP provides the state-of-the-art results and resolves the failures of previous approaches by eliminating all of the predefined situations and constraints of applying perspective projection as the camera model.

[1]  Francesc Moreno-Noguer,et al.  Sequential Non-Rigid Structure from Motion Using Physical Priors , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Shohreh Kasaei,et al.  3D reconstruction of non-rigid surfaces from realistic monocular video , 2015, 2015 9th Iranian Conference on Machine Vision and Image Processing (MVIP).

[3]  Henning Biermann,et al.  Recovering non-rigid 3D shape from image streams , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[4]  Francesc Moreno-Noguer,et al.  Simultaneous pose and non-rigid shape with particle dynamics , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Aaron Hertzmann,et al.  Learning Non-Rigid 3D Shape from 2D Motion , 2003, NIPS.

[6]  Lourdes Agapito,et al.  Soft Inextensibility Constraints for Template-Free Non-rigid Reconstruction , 2012, ECCV.

[7]  Rui Yu,et al.  Direct, Dense, and Deformable: Template-Based Non-rigid 3D Reconstruction from RGB Video , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[8]  Geonho Cha,et al.  Non-rigid surface recovery with a robust local-rigidity prior , 2018, Pattern Recognit. Lett..

[9]  Suryansh Kumar,et al.  Non-Rigid Structure from Motion: Prior-Free Factorization Method Revisited , 2019, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).

[10]  Matthew Brand,et al.  A direct method for 3D factorization of nonrigid motion observed in 2D , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[11]  Henrik Aanæs,et al.  Estimation of Deformable Structure and Motion , 2002 .

[12]  Luis Miguel Bergasa,et al.  Model-Based Real-Time Non-Rigid Tracking , 2017, Sensors.

[13]  Aleix M. Martínez,et al.  Computing Smooth Time Trajectories for Camera and Deformable Shape in Structure from Motion with Occlusion , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Jitendra Malik,et al.  Grouping-Based Low-Rank Trajectory Completion and 3D Reconstruction , 2014, NIPS.

[15]  Luc Van Gool,et al.  Incremental Non-Rigid Structure-from-Motion with Unknown Focal Length , 2018, ECCV.

[16]  Takeo Kanade,et al.  Nonrigid Structure from Motion in Trajectory Space , 2008, NIPS.

[17]  Lourdes Agapito,et al.  A Variational Approach to Video Registration with Subspace Constraints , 2013, International Journal of Computer Vision.

[18]  Pascal Fua,et al.  Template-free monocular reconstruction of deformable surfaces , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[19]  Yaming Wang,et al.  Non-Rigid Structure Estimation in Trajectory Space from Monocular Vision , 2015, Sensors.

[20]  Adrien Bartoli,et al.  Inextensible Non-Rigid Structure-from-Motion by Second-Order Cone Programming , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Jing Xiao,et al.  A Closed-Form Solution to Non-Rigid Shape and Motion Recovery , 2004, International Journal of Computer Vision.

[22]  Aaron Hertzmann,et al.  Nonrigid Structure-from-Motion: Estimating Shape and Motion with Hierarchical Priors , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Alessio Del Bue,et al.  Optimal Metric Projections for Deformable and Articulated Structure-from-Motion , 2011, International Journal of Computer Vision.

[24]  Alessio Del Bue,et al.  Non-Rigid Metric Shape and Motion Recovery from Uncalibrated Images Using Priors , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[25]  Rui Yu,et al.  Video Pop-up: Monocular 3D Reconstruction of Dynamic Scenes , 2014, ECCV.

[26]  Hongdong Li,et al.  A Simple Prior-Free Method for Non-rigid Structure-from-Motion Factorization , 2012, International Journal of Computer Vision.

[27]  Lourdes Agapito,et al.  Good Vibrations: A Modal Analysis Approach for Sequential Non-rigid Structure from Motion , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[28]  Fei Wang,et al.  Template-Free 3D Reconstruction of Poorly-Textured Nonrigid Surfaces , 2016, ECCV.

[29]  Daniel Pizarro-Perez,et al.  Self-Calibrating Isometric Non-Rigid Structure-from-Motion , 2018, ECCV.

[30]  René Vidal,et al.  Perspective Nonrigid Shape and Motion Recovery , 2008, ECCV.

[31]  Takeo Kanade,et al.  Trajectory Space: A Dual Representation for Nonrigid Structure from Motion , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Hongdong Li,et al.  Spatial-Temporal Union of Subspaces for Multi-body Non-rigid Structure-from-Motion , 2017, ArXiv.

[33]  Jing Xiao,et al.  Uncalibrated perspective reconstruction of deformable structures , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[34]  Takeo Kanade,et al.  Shape and motion from image streams under orthography: a factorization method , 1992, International Journal of Computer Vision.

[35]  Xiaomeng Yan,et al.  Research on Non-rigid Structure from Motion: A Literature Review , 2015 .

[36]  Sohaib Khan,et al.  NRSfM using local rigidity , 2014, IEEE Winter Conference on Applications of Computer Vision.

[37]  Adrien Bartoli,et al.  Non-Rigid Shape-from-Motion for Isometric Surfaces using Infinitesimal Planarity , 2014, BMVC.

[38]  S. Kasaei,et al.  Perspective reconstruction of non-rigid surfaces from single-view videos , 2017, 2017 Iranian Conference on Electrical Engineering (ICEE).

[39]  Lourdes Agapito,et al.  Dense Variational Reconstruction of Non-rigid Surfaces from Monocular Video , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[40]  Yaser Sheikh,et al.  Separable Spatiotemporal Priors for Convex Reconstruction of Time-Varying 3D Point Clouds , 2014, ECCV.

[41]  Kiriakos N. Kutulakos,et al.  Non-rigid structure from locally-rigid motion , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.