Dense Image Registration and Deformable Surface Reconstruction in Presence of Occlusions and Minimal Texture

Deformable surface tracking from monocular images is well-known to be under-constrained. Occlusions often make the task even more challenging, and can result in failure if the surface is not sufficiently textured. In this work, we explicitly address the problem of 3D reconstruction of poorly textured, occluded surfaces, proposing a framework based on a template-matching approach that scales dense robust features by a relevancy score. Our approach is extensively compared to current methods employing both local feature matching and dense template alignment. We test on standard datasets as well as on a new dataset (that will be made publicly available) of a sparsely textured, occluded surface. Our framework achieves state-of-the-art results for both well and poorly textured, occluded surfaces.

[1]  Vincent Lepetit,et al.  Retexturing in the Presence of Complex Illumination and Occlusions , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[2]  K. V. Arya,et al.  Image registration using robust M-estimators , 2007, Pattern Recognit. Lett..

[3]  Geraldo F. Silveira,et al.  Real-time Visual Tracking under Arbitrary Illumination Changes , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  Alois Knoll,et al.  Mutual Information-Based 3D Object Tracking , 2008, International Journal of Computer Vision.

[5]  Raquel Urtasun,et al.  Beyond Feature Points: Structured Prediction for Monocular Non-rigid 3D Reconstruction , 2012, ECCV.

[6]  Pascal Fua,et al.  Linear Local Models for Monocular Reconstruction of Deformable Surfaces , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Florent Brunet,et al.  Monocular template-based 3D surface reconstruction: Convex inextensible and nonconvex isometric methods , 2014, Comput. Vis. Image Underst..

[8]  Toby Sharp,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR.

[9]  Daniel Pizarro-Perez,et al.  Feature-Based Deformable Surface Detection with Self-Occlusion Reasoning , 2011, International Journal of Computer Vision.

[10]  Pascal Fua,et al.  A constrained latent variable model , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Éric Marchand,et al.  Second-Order Optimization of Mutual Information for Real-Time Image Registration , 2012, IEEE Transactions on Image Processing.

[12]  Fred L. Bookstein,et al.  Principal Warps: Thin-Plate Splines and the Decomposition of Deformations , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  B. Ripley,et al.  Robust Statistics , 2018, Encyclopedia of Mathematical Geosciences.

[14]  Adrien Bartoli,et al.  Using Isometry to Classify Correct/Incorrect 3D-2D Correspondences , 2014, ECCV.

[15]  David W. Jacobs,et al.  Computer Vision and Image Understanding 114 (2010) 135–145 Contents lists available at ScienceDirect Computer Vision and Image Understanding , 2022 .

[16]  Adrien Bartoli,et al.  On template-based reconstruction from a single view: Analytical solutions and proofs of well-posedness for developable, isometric and conformal surfaces , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Dima Damen,et al.  Real-time Learning and Detection of 3D Texture-less Objects: A Scalable Approach , 2012, BMVC 2012.

[18]  F. Mosteller,et al.  Understanding robust and exploratory data analysis , 1985 .

[19]  Adrien Bartoli,et al.  Monocular Template-based Reconstruction of Inextensible Surfaces , 2011, International Journal of Computer Vision.

[20]  Shuicheng Yan,et al.  An HOG-LBP human detector with partial occlusion handling , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[21]  Richard Bowden,et al.  A Unifying Framework for Mutual Information Methods for Use in Non-linear Optimisation , 2006, ECCV.

[22]  G. Klein,et al.  Parallel Tracking and Mapping for Small AR Workspaces , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[23]  Pascal Fua,et al.  Laplacian Meshes for Monocular 3D Shape Recovery , 2012, ECCV.

[24]  Peter J. Huber,et al.  Robust Statistics , 2005, Wiley Series in Probability and Statistics.

[25]  Hossein Mobahi,et al.  Face recognition with contiguous occlusion using markov random fields , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[26]  Luc Van Gool,et al.  Combined Depth and Outlier Estimation in Multi-View Stereo , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[27]  Lourdes Agapito,et al.  Dense Non-rigid Structure from Motion , 2012, 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission.

[28]  Lourdes Agapito,et al.  Real-time sequential model-based non-rigid SFM , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[29]  Nassir Navab,et al.  Monocular Template-Based Reconstruction of Smooth and Inextensible Surfaces , 2010, ACCV.

[30]  Pascal Fua,et al.  Local deformation models for monocular 3D shape recovery , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[31]  Lourdes Agapito,et al.  Soft Inextensibility Constraints for Template-Free Non-rigid Reconstruction , 2012, ECCV.

[32]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[33]  Gregory D. Hager,et al.  Efficient Region Tracking With Parametric Models of Geometry and Illumination , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[34]  Adrien Bartoli,et al.  A pixel-based approach to template-based monocular 3D reconstruction of deformable surfaces , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[35]  Maxime Meilland,et al.  Improving NCC-Based Direct Visual Tracking , 2012, ECCV.

[36]  J. P. Lewis.  Fast Normalized Cross-Correlation , 2010 .

[37]  Paul A. Viola,et al.  Alignment by Maximization of Mutual Information , 1997, International Journal of Computer Vision.

[38]  Mathieu Salzmann,et al.  Continuous Inference in Graphical Models with Polynomial Energies , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[39]  Vincent Lepetit,et al.  Robust 3D Tracking with Descriptor Fields , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[40]  Adrien Bartoli,et al.  Stable Template-Based Isometric 3D Reconstruction in All Imaging Conditions by Linear Least-Squares , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[41]  Takeo Kanade,et al.  Trajectory Space: A Dual Representation for Nonrigid Structure from Motion , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Lourdes Agapito,et al.  Dense Variational Reconstruction of Non-rigid Surfaces from Monocular Video , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.