Handling Occlusions and Sparse Textures in a Deformable Surface Tracking Framework

Deformable surface tracking from monocular images is a well-known under-constrained problem. Occlusions often make the task even more challenging, and can lead current methods to fail if the surface is not sufficiently textured. In this work, we explicitly address the problem of 3D reconstruction of poorly textured, occluded surfaces, proposing a framework based on a template-matching approach that scales dense robust features by a relevancy score. Our approach is extensively compared to current methods employing both local feature matching and dense template alignment. We test on standard datasets as well as on a new dataset (that will be made publicly available) of a sparsely textured, occluded surface. Our framework achieves state-of-the-art results for both well and poorly textured, occluded surfaces.

[1]  Daniel Pizarro-Perez,et al.  Feature-Based Deformable Surface Detection with Self-Occlusion Reasoning , 2011, International Journal of Computer Vision.

[2]  Luc Van Gool,et al.  Combined Depth and Outlier Estimation in Multi-View Stereo , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[3]  Lourdes Agapito,et al.  Dense Non-rigid Structure from Motion , 2012, 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission.

[4]  Fred L. Bookstein,et al.  Principal Warps: Thin-Plate Splines and the Decomposition of Deformations , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[6]  Lourdes Agapito,et al.  Dense Variational Reconstruction of Non-rigid Surfaces from Monocular Video , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Shuicheng Yan,et al.  An HOG-LBP human detector with partial occlusion handling , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[8]  Gregory D. Hager,et al.  Efficient Region Tracking With Parametric Models of Geometry and Illumination , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[9]  Richard Bowden,et al.  A Unifying Framework for Mutual Information Methods for Use in Non-linear Optimisation , 2006, ECCV.

[10]  Mathieu Salzmann,et al.  Continuous Inference in Graphical Models with Polynomial Energies , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Vincent Lepetit,et al.  Robust 3D Tracking with Descriptor Fields , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  B. Ripley,et al.  Robust Statistics , 2018, Wiley Series in Probability and Statistics.

[13]  Geraldo F. Silveira,et al.  Real-time Visual Tracking under Arbitrary Illumination Changes , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Dima Damen,et al.  Real-time Learning and Detection of 3D Texture-less Objects: A Scalable Approach , 2012, BMVC 2012.

[15]  Maxime Meilland,et al.  Improving NCC-Based Direct Visual Tracking , 2012, ECCV.

[16]  Vincent Lepetit,et al.  Retexturing in the Presence of Complex Illumination and Occlusions , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[17]  Pascal Fua,et al.  Laplacian Meshes for Monocular 3D Shape Recovery , 2012, ECCV.

[18]  K. V. Arya,et al.  Image registration using robust M-estimators , 2007, Pattern Recognit. Lett..

[19]  Pascal Fua,et al.  Local deformation models for monocular 3D shape recovery , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  Hossein Mobahi,et al.  Face recognition with contiguous occlusion using markov random fields , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[21]  Pascal Fua,et al.  Linear Local Models for Monocular Reconstruction of Deformable Surfaces , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Adrien Bartoli,et al.  A pixel-based approach to template-based monocular 3D reconstruction of deformable surfaces , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[23]  Nassir Navab,et al.  Monocular Template-Based Reconstruction of Smooth and Inextensible Surfaces , 2010, ACCV.

[24]  Lourdes Agapito,et al.  Soft Inextensibility Constraints for Template-Free Non-rigid Reconstruction , 2012, ECCV.

[25]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[26]  Lourdes Agapito,et al.  Real-time sequential model-based non-rigid SFM , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[27]  F. Mosteller,et al.  Understanding robust and exploratory data analysis , 1985 .

[28]  Adrien Bartoli,et al.  Monocular Template-based Reconstruction of Inextensible Surfaces , 2011, International Journal of Computer Vision.

[29]  Pascal Fua,et al.  A constrained latent variable model , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[30]  Éric Marchand,et al.  Second-Order Optimization of Mutual Information for Real-Time Image Registration , 2012, IEEE Transactions on Image Processing.

[31]  Adrien Bartoli,et al.  Stable Template-Based Isometric 3D Reconstruction in All Imaging Conditions by Linear Least-Squares , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[32]  Takeo Kanade,et al.  Trajectory Space: A Dual Representation for Nonrigid Structure from Motion , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Alois Knoll,et al.  Mutual Information-Based 3D Object Tracking , 2008, International Journal of Computer Vision.

[34]  Raquel Urtasun,et al.  Beyond Feature Points: Structured Prediction for Monocular Non-rigid 3D Reconstruction , 2012, ECCV.

[35]  David W. Jacobs,et al.  Computer Vision and Image Understanding 114 (2010) 135–145 Contents lists available at ScienceDirect Computer Vision and Image Understanding , 2022 .

[36]  Adrien Bartoli,et al.  On template-based reconstruction from a single view: Analytical solutions and proofs of well-posedness for developable, isometric and conformal surfaces , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[37]  J. P. Lewis.  Fast Normalized Cross-Correlation , 2010 .

[38]  Paul A. Viola,et al.  Alignment by Maximization of Mutual Information , 1997, International Journal of Computer Vision.