Model-Based Real-Time Non-Rigid Tracking

This paper presents a sequential non-rigid reconstruction method that recovers the 3D shape and the camera pose of a deforming object from a video sequence and a previous shape model of the object. We take PTAM (Parallel Mapping and Tracking), a state-of-the-art sequential real-time SfM (Structure-from-Motion) engine, and we upgrade it to solve non-rigid reconstruction. Our method provides a good trade-off between processing time and reconstruction error without the need for specific processing hardware, such as GPUs. We improve the original PTAM matching by using descriptor-based features, as well as smoothness priors to better constrain the 3D error. This paper works with perspective projection and deals with outliers and missing data. We evaluate the tracking algorithm performance through different tests over several datasets of non-rigid deforming objects. Our method achieves state-of-the-art accuracy and can be used as a real-time method suitable for being embedded in portable devices.

[1]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[2]  Dieter Fox,et al.  DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Rui Yu,et al.  Video Pop-up: Monocular 3D Reconstruction of Dynamic Scenes , 2014, ECCV.

[4]  Lourdes Agapito,et al.  Factorization for non-rigid and articulated structure using metric projections , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[6]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[7]  Aaron Hertzmann,et al.  Learning Non-Rigid 3D Shape from 2D Motion , 2003, NIPS.

[8]  Lourdes Agapito,et al.  Online Dense Non-Rigid 3D Shape and Camera Motion Recovery , 2014, BMVC.

[9]  J. M. M. Montiel,et al.  Finite Element based sequential Bayesian Non-Rigid Structure from Motion , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Mathieu Salzmann,et al.  Continuous Inference in Graphical Models with Polynomial Energies , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[12]  Richard Szeliski,et al.  Building Rome in a day , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[13]  Aaron Hertzmann,et al.  Nonrigid Structure-from-Motion: Estimating Shape and Motion with Hierarchical Priors , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Enrique Muñoz,et al.  A direct approach for efficiently tracking with 3D morphable models , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[15]  Daniel Pizarro-Perez,et al.  Feature-Based Deformable Surface Detection with Self-Occlusion Reasoning , 2011, International Journal of Computer Vision.

[16]  Adrien Bartoli,et al.  Template-Based Isometric Deformable 3D Reconstruction with Sampling-Based Focal Length Self-Calibration , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Daniel Pizarro-Perez,et al.  Shape-from-Template , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Gary R. Bradski,et al.  ORB: An efficient alternative to SIFT or SURF , 2011, 2011 International Conference on Computer Vision.

[19]  Christopher Hunt,et al.  Notes on the OpenSURF Library , 2009 .

[20]  Nassir Navab,et al.  Monocular Template-Based Reconstruction of Smooth and Inextensible Surfaces , 2010, ACCV.

[21]  Lourdes Agapito,et al.  Robust Trajectory-Space TV-L1 Optical Flow for Non-rigid Sequences , 2011, EMMCVPR.

[22]  Lourdes Agapito,et al.  Soft Inextensibility Constraints for Template-Free Non-rigid Reconstruction , 2012, ECCV.

[23]  Tom Drummond,et al.  Machine Learning for High-Speed Corner Detection , 2006, ECCV.

[24]  Lourdes Agapito,et al.  Dense Variational Reconstruction of Non-rigid Surfaces from Monocular Video , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  Marinos Ioannides,et al.  In the wild image retrieval and clustering for 3D cultural heritage landmarks reconstruction , 2014, Multimedia Tools and Applications.

[26]  Pascal Fua,et al.  Template-Based Monocular 3D Shape Recovery Using Laplacian Meshes , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Henning Biermann,et al.  Recovering non-rigid 3D shape from image streams , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[28]  Daniel Pizarro-Perez,et al.  Inextensible Non-Rigid Shape-from-Motion by Second-Order Cone Programming , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  G. Klein,et al.  Parallel Tracking and Mapping for Small AR Workspaces , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[30]  Roland Siegwart,et al.  BRISK: Binary Robust invariant scalable keypoints , 2011, 2011 International Conference on Computer Vision.

[31]  Lourdes Agapito,et al.  Real-time sequential model-based non-rigid SFM , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[32]  K. Wenzel,et al.  4D Reconstruction of Tangible Cultural Heritage Objects from Web-Retrieved Images , 2014 .

[33]  Daniel Pizarro-Perez,et al.  Computer-Assisted Laparoscopic myomectomy by augmenting the uterus with pre-operative MRI data , 2014, 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[34]  Francesc Moreno-Noguer,et al.  Sequential Non-Rigid Structure from Motion Using Physical Priors , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Adrien Bartoli,et al.  Stable Template-Based Isometric 3D Reconstruction in All Imaging Conditions by Linear Least-Squares , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  René Vidal,et al.  Perspective Nonrigid Shape and Motion Recovery , 2008, ECCV.

[37]  Francesc Moreno-Noguer,et al.  Real-time 3D reconstruction of non-rigid shapes with a single moving camera , 2016, Comput. Vis. Image Underst..

[38]  Olivier Stasse,et al.  MonoSLAM: Real-Time Single Camera SLAM , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[39]  Frank Dellaert,et al.  Square Root SAM: Simultaneous Localization and Mapping via Square Root Information Smoothing , 2006, Int. J. Robotics Res..

[40]  Adrien Bartoli,et al.  Coarse-to-fine low-rank structure-from-motion , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[41]  Adrien Bartoli,et al.  Sequential Non-Rigid Structure-from-Motion with the 3D-Implicit Low-Rank Shape Model , 2010, ECCV.

[42]  Adrien Bartoli,et al.  Non-Rigid Shape-from-Motion for Isometric Surfaces using Infinitesimal Planarity , 2014, BMVC.

[43]  Lourdes Agapito,et al.  Energy based multiple model fitting for non-rigid structure from motion , 2011, CVPR 2011.

[44]  Pascal Fua,et al.  Monocular 3D Reconstruction of Locally Textured Surfaces , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[45]  Andrew Zisserman,et al.  Multiple View Geometry in Computer Vision (2nd ed) , 2003 .

[46]  Stefanos Zafeiriou,et al.  A 3D Morphable Model Learnt from 10,000 Faces , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[47]  Francesc Moreno-Noguer,et al.  Probabilistic simultaneous pose and non-rigid shape recovery , 2011, CVPR 2011.

[48]  David W. Murray,et al.  Parallel Tracking and Mapping on a camera phone , 2009, 2009 8th IEEE International Symposium on Mixed and Augmented Reality.

[49]  Snigdhansu Chatterjee,et al.  Procrustes Problems , 2005, Technometrics.

[50]  Adrien Bartoli,et al.  Realtime Shape-from-Template: System and Applications. , 2015, ISMAR 2015.

[51]  J. M. M. Montiel,et al.  3D Reconstruction of Non-Rigid Surfaces in Real-Time Using Wedge Elements , 2012, ECCV Workshops.

[52]  Lena Maier-Hein,et al.  Comparative Validation of Single-Shot Optical Techniques for Laparoscopic 3-D Surface Reconstruction , 2014, IEEE Transactions on Medical Imaging.

[53]  Francesc Moreno-Noguer,et al.  DaLI: Deformation and Light Invariant Descriptor , 2015, International Journal of Computer Vision.

[54]  Takeo Kanade,et al.  Nonrigid Structure from Motion in Trajectory Space , 2008, NIPS.

[55]  Songhwai Oh,et al.  Consensus of Non-rigid Reconstructions , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  Pascal Fua,et al.  A constrained latent variable model , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[57]  Hongdong Li,et al.  A Simple Prior-Free Method for Non-rigid Structure-from-Motion Factorization , 2012, International Journal of Computer Vision.

[58]  Adrien Bartoli,et al.  Isometric Non-rigid Shape-from-Motion in Linear Time , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[59]  Alessio Del Bue,et al.  Piecewise Quadratic Reconstruction of Non-Rigid Surfaces from Monocular Sequences , 2010, ECCV.

[60]  Selim Benhimane,et al.  Homography-based 2D Visual Tracking and Servoing , 2007, Int. J. Robotics Res..

[61]  Andrew J. Davison,et al.  DTAM: Dense tracking and mapping in real-time , 2011, 2011 International Conference on Computer Vision.

[62]  Kurt Keutzer,et al.  Dense Point Trajectories by GPU-Accelerated Large Displacement Optical Flow , 2010, ECCV.

[63]  Alessio Del Bue,et al.  Non-Rigid Metric Shape and Motion Recovery from Uncalibrated Images Using Priors , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[64]  Lourdes Agapito,et al.  Good Vibrations: A Modal Analysis Approach for Sequential Non-rigid Structure from Motion , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[65]  Charles Elkan,et al.  Expectation Maximization Algorithm , 2010, Encyclopedia of Machine Learning.

[66]  Adrien Bartoli,et al.  KAZE Features , 2012, ECCV.

[67]  Rui Yu,et al.  Direct, Dense, and Deformable: Template-Based Non-rigid 3D Reconstruction from RGB Video , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[68]  Peter F. Sturm,et al.  A Factorization Based Algorithm for Multi-Image Projective Structure and Motion , 1996, ECCV.

[69]  Stefanos Zafeiriou,et al.  Estimating Correspondences of Deformable Objects “In-the-Wild” , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[70]  Aleix M. Martínez,et al.  Non-rigid structure from motion with complementary rank-3 spaces , 2011, CVPR 2011.

[71]  Adrien Bartoli,et al.  Fast Explicit Diffusion for Accelerated Features in Nonlinear Scale Spaces , 2013, BMVC.

[72]  Nassir Navab,et al.  Deformable Template Tracking in 1ms , 2014, BMVC.

[73]  Pascal Fua,et al.  Linear Local Models for Monocular Reconstruction of Deformable Surfaces , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.