Shape-From-Template with Curves

Shape-from-Template (SfT) is the problem of using a shape template to infer the shape of a deformable object observed in an image. The usual case of SfT is ‘Surface’ SfT, where the shape is a 2D surface embedded in 3D, and the image is a 2D perspective projection. We introduce ‘Curve’ SfT, comprising two new cases of SfT where the shape is a 1D curve. The first new case is when the curve is embedded in 2D and the image a 1D perspective projection. The second new case is when the curve is embedded in 3D and the image a 2D perspective projection. We present a thorough theoretical study of these new cases for isometric deformations, which are a good approximation of ropes, cables and wires. Unlike Surface SfT, we show that Curve SfT is only ever solvable up to discrete ambiguities. We present the necessary and sufficient conditions for solvability with critical point analysis. We further show that unlike Surface SfT, Curve SfT cannot be solved locally using exact non-holonomic Partial Differential Equations. Our main technical contributions are two-fold. First, we give a stable, global reconstruction method that models the problem as a discrete Hidden Markov Model. This can generate all candidate solutions. Second, we give a non-convex refinement method using a novel angle-based deformation parameterization. We present quantitative and qualitative results showing that real curve shaped objects such as a necklace can be successfully reconstructed with Curve SfT.

[1]  Jos F. Sturm,et al.  A Matlab toolbox for optimization over symmetric cones , 1999 .

[2]  Andres Fco. Solé,et al.  3D Curves Reconstruction Based on Deformable Models , 2003, Journal of Mathematical Imaging and Vision.

[3]  Adrien Bartoli,et al.  Combining Conformal Deformation and Cook–Torrance Shading for 3-D Reconstruction in Laparoscopy , 2014, IEEE Transactions on Biomedical Engineering.

[4]  Rui Yu,et al.  Direct, Dense, and Deformable: Template-Based Non-rigid 3D Reconstruction from RGB Video , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[5]  Adrien Bartoli,et al.  A pixel-based approach to template-based monocular 3D reconstruction of deformable surfaces , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[6]  Yeung Sam Hung,et al.  3D Curves Reconstruction from Multiple Images , 2010, 2010 International Conference on Digital Image Computing: Techniques and Applications.

[7]  Andrew Zisserman,et al.  Multiple View Geometry in Computer Vision (2nd ed) , 2003 .

[8]  Adrien Bartoli,et al.  Energy-Based Reconstruction of 3D Curves for Quality Control , 2007, EMMCVPR.

[9]  Adrien Bartoli,et al.  Monocular Template-Based 3D Reconstruction of Extensible Surfaces with Local Linear Elasticity , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  J. Lofberg,et al.  YALMIP : a toolbox for modeling and optimization in MATLAB , 2004, 2004 IEEE International Conference on Robotics and Automation (IEEE Cat. No.04CH37508).

[11]  Adrien Bartoli,et al.  A linear least-squares solution to elastic Shape-from-Template , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Olivier D. Faugeras,et al.  A theory of the motion fields of curves , 1993, International Journal of Computer Vision.

[13]  Anders Heyden,et al.  Reconstruction of General Curves, Using Factorization and Bundle Adjustment , 2004, International Journal of Computer Vision.

[14]  Pascal Fua,et al.  Template-Based Monocular 3D Shape Recovery Using Laplacian Meshes , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[16]  Pascal Fua,et al.  Dense Image Registration and Deformable Surface Reconstruction in Presence of Occlusions and Minimal Texture , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[17]  Daniel Pizarro-Perez,et al.  Solutions of Quadratic First-Order ODEs applied to Computer Vision Problems , 2017, ArXiv.

[18]  Daniel Pizarro-Perez,et al.  Feature-Based Deformable Surface Detection with Self-Occlusion Reasoning , 2011, International Journal of Computer Vision.

[19]  Fredrik Kahl,et al.  Multiview reconstruction of space curves , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[20]  Nazim Haouchine,et al.  Single view augmentation of 3D elastic objects , 2014, 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[21]  Adrien Bartoli,et al.  Using Shading and a 3D Template to Reconstruct Complex Surface Deformations , 2016, BMVC.

[22]  Andrew W. Fitzgibbon,et al.  Better Together: Joint Reasoning for Non-rigid 3D Reconstruction with Specularities and Shading , 2017, BMVC.

[23]  Adrien Bartoli,et al.  [POSTER] Realtime Shape-from-Template: System and Applications , 2015, 2015 IEEE International Symposium on Mixed and Augmented Reality.

[24]  Adrien Bartoli,et al.  Realtime Shape-from-Template: System and Applications. , 2015, ISMAR 2015.

[25]  Thomas Vetter,et al.  A morphable model for the synthesis of 3D faces , 1999, SIGGRAPH.

[26]  Adrien Bartoli,et al.  An Analysis of Errors in Graph-Based Keypoint Matching and Proposed Solutions , 2014, ECCV.

[27]  Daniel Pizarro-Perez,et al.  Shape-from-Template , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Yakov Eliashberg,et al.  Introduction to the h-Principle , 2002 .

[29]  Adrien Bartoli,et al.  Robust, Real-Time, Dense and Deformable 3D Organ Tracking in Laparoscopic Videos , 2016, MICCAI.

[30]  Pascal Fua,et al.  Reconstructing sharply folding surfaces: A convex formulation , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[31]  Adrien Bartoli,et al.  Particle-SfT: A Provably-Convergent, Fast Shape-from-Template Algorithm , 2017, International Journal of Computer Vision.

[32]  Olivier D. Faugeras,et al.  Curve-based stereo: figural continuity and curvature , 1991, Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[33]  Adrien Bartoli,et al.  As-Rigid-as-Possible Volumetric Shape-from-Template , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[34]  Adrien Bartoli,et al.  A Perspective on Non-Isometric Shape-from-Template , 2016, 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct).

[35]  Adrien Bartoli,et al.  Using Isometry to Classify Correct/Incorrect 3D-2D Correspondences , 2014, ECCV.

[36]  Matthew Turk,et al.  A Morphable Model For The Synthesis Of 3D Faces , 1999, SIGGRAPH.

[37]  Adrien Bartoli,et al.  Shape-from-Template in Flatland , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Johan Efberg,et al.  YALMIP : A toolbox for modeling and optimization in MATLAB , 2004 .

[39]  Marc Alexa,et al.  As-rigid-as-possible surface modeling , 2007, Symposium on Geometry Processing.

[40]  Lourdes Agapito,et al.  Balloon Shapes: Reconstructing and Deforming Objects with Volume from Images , 2013, 2013 International Conference on 3D Vision.

[41]  Pascal Fua,et al.  Linear Local Models for Monocular Reconstruction of Deformable Surfaces , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Adrien Bartoli,et al.  Monocular Template-based Reconstruction of Inextensible Surfaces , 2011, International Journal of Computer Vision.

[43]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[44]  Adrien Bartoli,et al.  A Stable Analytical Framework for Isometric Shape-from-Template by Surface Integration , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[45]  Pascal Fua,et al.  Surface Deformation Models for Nonrigid 3D Shape Recovery , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  Florent Brunet,et al.  Monocular template-based 3D surface reconstruction: Convex inextensible and nonconvex isometric methods , 2014, Comput. Vis. Image Underst..