Three-Dimensional Shape Knowledge for Joint Image Segmentation and Pose Estimation

This paper presents the integration of 3D shape knowledge into a variational model for level set based image segmentation and tracking. Having a 3D surface model of an object that is visible in the image of a calibrated camera, the object contour stemming from the segmentation is applied to estimate the 3D pose parameters, whereas the object model projected to the image plane helps in a top-down manner to improve the extraction of the contour and the region statistics. The present approach clearly states all model assumptions in a single energy functional. This keeps the model manageable and allows further extensions for the future. While common alternative segmentation approaches that integrate 2D shape knowledge face the problem that an object can look very different from various viewpoints, a 3D free form model ensures that for each view the model can perfectly fit the data in the image. Moreover, one solves the higher level problem of determining the object pose including its distance to the camera. Experiments demonstrate the performance of the method.

[1]  S. Shankar Sastry,et al.  An Invitation to 3-D Vision , 2004 .

[2]  Daniel Cremers,et al.  Motion Competition: A variational framework for piecewise parametric motion segmentation , 2005 .

[3]  John W. Fisher,et al.  Nonparametric methods for image segmentation using information theory and curve evolution , 2002, Proceedings. International Conference on Image Processing.

[4]  Daniel Cremers,et al.  Multiphase Dynamic Labeling for Variational Recognition-Driven Image Segmentation , 2004, ECCV.

[5]  Nahum Kiryati,et al.  Unlevel-Sets: Geometry and Prior-Based Segmentation , 2004, ECCV.

[6]  John W. Fisher,et al.  Submitted to Ieee Transactions on Image Processing a Nonparametric Statistical Method for Image Segmentation Using Information Theory and Curve Evolution , 2022 .

[7]  Rachid Deriche,et al.  Implicit Active Shape Models for 3D Segmentation in MR Imaging , 2004, MICCAI.

[8]  Rachid Deriche,et al.  Unifying boundary and region-based information for geodesic active tracking , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[9]  J. Beveridge.  Local search algorithms for geometric object recognition: optimal correspondence and pose , 1993 .

[10]  Baba C. Vemuri,et al.  Shape Modeling with Front Propagation: A Level Set Approach , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[11]  Jitendra Malik,et al.  Contour and Texture Analysis for Image Segmentation , 2001, International Journal of Computer Vision.

[12]  Fergal Shevlin,et al.  Analysis of orientation problems using Plucker lines , 1998, Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No.98EX170).

[13]  Paul J. Besl,et al.  The Free-Form Surface Matching Problem , 1990 .

[14]  A. Yezzi,et al.  A variational framework for joint segmentation and registration , 2001, Proceedings IEEE Workshop on Mathematical Methods in Biomedical Image Analysis (MMBIA 2001).

[15]  David G. Lowe,et al.  Three-Dimensional Object Recognition from Single Two-Dimensional Images , 1987, Artif. Intell..

[16]  D. Cremers,et al.  Diffusion-snakes: combining statistical shape knowledge and image information in a variational framework , 2001, Proceedings IEEE Workshop on Variational and Level Set Methods in Computer Vision.

[17]  Tony F. Chan,et al.  A Multiphase Level Set Framework for Image Segmentation Using the Mumford and Shah Model , 2002, International Journal of Computer Vision.

[18]  Mongi A. Abidi,et al.  Pose and motion estimation from vision using dual quaternion-based extended kalman filtering , 1997 .

[19]  O. Faugeras,et al.  Variational principles, surface evolution, PDE's, level set methods and the stereo problem , 1998, 5th IEEE EMBS International Summer School on Biomedical Imaging, 2002..

[20]  Bodo Rosenhahn,et al.  Pose estimation revisited , 2006 .

[21]  Nikos Paragios,et al.  Non-rigid registration using distance functions , 2003, Comput. Vis. Image Underst..

[22]  T. Chan,et al.  A Variational Level Set Approach to Multiphase Motion , 1996 .

[23]  Nikos Paragios,et al.  Shape Priors for Level Set Representations , 2002, ECCV.

[24]  Patrick Bouthemy,et al.  A 2D-3D model-based approach to real-time visual tracking , 2001, Image Vis. Comput..

[25]  J. Sethian,et al.  Fronts propagating with curvature-dependent speed: algorithms based on Hamilton-Jacobi formulations , 1988 .

[26]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[27]  O. Faugeras,et al.  Statistical shape influence in geodesic active contours , 2002, 5th IEEE EMBS International Summer School on Biomedical Imaging, 2002..

[28]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[29]  Rachid Deriche,et al.  Geodesic Active Regions and Level Set Methods for Supervised Texture Segmentation , 2002, International Journal of Computer Vision.

[30]  R. Deriche,et al.  A variational framework for active and adaptative segmentation of vector valued images , 2002, Workshop on Motion and Video Computing, 2002. Proceedings..

[31]  Thomas Brox,et al.  A TV flow based local scale estimate and its application to texture discrimination , 2006, J. Vis. Commun. Image Represent..

[32]  Demetri Terzopoulos,et al.  Snakes: Active contour models , 2004, International Journal of Computer Vision.

[33]  Bodo Rosenhahn,et al.  Pose Estimation of Free-Form Objects , 2004, ECCV.

[34]  Rachid Deriche,et al.  Active unsupervised texture segmentation on a diffusion based feature space , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[35]  D. Mumford,et al.  Optimal approximations by piecewise smooth functions and associated variational problems , 1989 .

[36]  Patrick J. Flynn,et al.  A Survey Of Free-Form Object Representation and Recognition Techniques , 2001, Comput. Vis. Image Underst..

[37]  R. Nevatia,et al.  Pose estimation of multi-part curved objects , 1995, Proceedings of International Symposium on Computer Vision - ISCV.

[38]  Michael Brady,et al.  Unsupervised non-parametric region segmentation using level sets , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[39]  Daniel Cremers,et al.  Motion Competition: A Variational Approach to Piecewise Parametric Motion Segmentation , 2005, International Journal of Computer Vision.

[40]  Thomas Brox,et al.  Universität Des Saarlandes Fachrichtung 6.1 – Mathematik Level Set Segmentation with Multiple Regions Level Set Segmentation with Multiple Regions , 2022 .

[41]  Daniel Cremers,et al.  Statistical shape knowledge in variational motion segmentation , 2003, Image Vis. Comput..

[42]  Helder Araújo,et al.  A Fully Projective Formulation to Improve the Accuracy of Lowe's Pose-Estimation Algorithm , 1998, Comput. Vis. Image Underst..

[43]  Jean Gallier,et al.  Geometric Methods and Applications: For Computer Science and Engineering , 2000 .

[44]  Stefano Soatto,et al.  Structure from motion for scenes without features , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[45]  Stan Z. Li,et al.  Markov Random Field Modeling in Computer Vision , 1995, Computer Science Workbench.

[46]  William Grimson,et al.  Object recognition by computer - the role of geometric constraints , 1991 .

[47]  O. Faugeras.  Three-dimensional computer vision: a geometric viewpoint , 1993 .

[48]  Georgios Tziritas,et al.  Bayesian Level Sets for Image Segmentation , 2002, J. Vis. Commun. Image Represent..

[49]  Thomas Brox,et al.  High Accuracy Optical Flow Estimation Based on a Theory for Warping , 2004, ECCV.

[50]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[51]  Daniel Cremers,et al.  Shape statistics in kernel space for variational image segmentation , 2003, Pattern Recognit..

[52]  Jitendra Malik,et al.  Tracking people with twists and exponential maps , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[53]  Alan L. Yuille,et al.  Region Competition: Unifying Snakes, Region Growing, and Bayes/MDL for Multiband Image Segmentation , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[54]  A. Dervieux,et al.  A finite element method for the simulation of a Rayleigh-Taylor instability , 1980 .

[55]  Gerald Sommer,et al.  Geometric Computing with Clifford Algebras , 2001, Springer Berlin Heidelberg.

[56]  Andrew Blake,et al.  Visual Reconstruction , 1987, Deep Learning for EEG-Based Brain–Computer Interfaces.

[57]  D. N. Prabhakar Murthy,et al.  Wiley Series in Probability and Statistics , 2003 .

[58]  David J. Kriegman,et al.  Constraints for Recognizing and Locating Curved 3D Objects from Monocular Image Features , 1992, ECCV.

[59]  Rachid Deriche,et al.  Geodesic active regions and level set methods for motion estimation and tracking , 2005, Comput. Vis. Image Underst..

[60]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[61]  Carlos Vázquez,et al.  Image partioning by level set multiregion competition , 2004, 2004 International Conference on Image Processing, 2004. ICIP '04..

[62]  Rachid Deriche,et al.  Dense Depth Map Reconstruction: A Minimization and Regularization Approach which Preserves Discontinuities , 1996, ECCV.

[63]  Daniel Cremers,et al.  Kernel Density Estimation and Intrinsic Alignment for Knowledge-Driven Segmentation: Teaching Level Sets to Walk , 2004, DAGM-Symposium.

[64]  Vincent Lepetit,et al.  Monocular Model-Based 3D Tracking of Rigid Objects: A Survey , 2005, Found. Trends Comput. Graph. Vis..

[65]  Jitendra Malik,et al.  Normalized Cuts and Image Segmentation , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[66]  Tony F. Chan,et al.  Active contours without edges , 2001, IEEE Trans. Image Process..

[67]  Daniel Cremers,et al.  Diffusion Snakes: Introducing Statistical Shape Knowledge into the Mumford-Shah Functional , 2002, International Journal of Computer Vision.

[68]  Bodo Rosenhahn,et al.  Three-Dimensional Shape Knowledge for Joint Image Segmentation and Pose Tracking , 2007, International Journal of Computer Vision.

[69]  Jitendra Malik,et al.  Twist Based Acquisition and Tracking of Animal and Human Kinematics , 2004, International Journal of Computer Vision.

[70]  Rachid Deriche,et al.  Geodesic Active Regions: A New Framework to Deal with Frame Partition Problems in Computer Vision , 2002, J. Vis. Commun. Image Represent..

[71]  Tony F. Chan,et al.  An Active Contour Model without Edges , 1999, Scale-Space.

[72]  Olivier D. Faugeras,et al.  Variational principles, surface evolution, PDEs, level set methods, and the stereo problem , 1998, IEEE Trans. Image Process..

[73]  Roberto Cipolla,et al.  Real-Time Tracking of Multiple Articulated Structures in Multiple Views , 2000, ECCV.

[74]  Zhengyou Zhang,et al.  Iterative point matching for registration of free-form curves and surfaces , 1994, International Journal of Computer Vision.

[75]  Thomas Brox,et al.  A TV Flow Based Local Scale Measure for Texture Discrimination , 2004, ECCV.

[76]  Anthony J. Yezzi,et al.  Curve evolution implementation of the Mumford-Shah functional for image segmentation, denoising, interpolation, and magnification , 2001, IEEE Trans. Image Process..

[77]  Vincent Lepetit,et al.  Stable real-time 3D tracking using online and offline information , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[78]  Hans-Hellmut Nagel,et al.  Combination of Edge Element and Optical Flow Estimates for 3D-Model-Based Vehicle Tracking in Traffic Image Sequences , 1999, International Journal of Computer Vision.

[79]  Christoph Schnörr,et al.  Natural Image Statistics for Natural Image Segmentation , 2005, International Journal of Computer Vision.

[80]  S. Shankar Sastry,et al.  A mathematical introduction to robotics manipulation , 1994 .

[81]  V. Caselles,et al.  A geometric model for active contours in image processing , 1993 .

[82]  Rachid Deriche,et al.  Unsupervised Segmentation Incorporating Colour, Texture, and Motion , 2003, CAIP.