Multibody Structure-from-Motion in Practice

Multibody structure from motion (SfM) is the extension of classical SfM to dynamic scenes with multiple rigidly moving objects. Recent research has unveiled some of the mathematical foundations of the problem, but a practical algorithm which can handle realistic sequences is still missing. In this paper, we discuss the requirements for such an algorithm, highlight theoretical issues and practical problems, and describe how a static structure-from-motion framework needs to be extended to handle real dynamic scenes. Theoretical issues include different situations in which the number of independently moving scene objects changes: Moving objects can enter or leave the field of view, merge into the static background (e.g., when a car is parked), or split off from the background and start moving independently. Practical issues arise due to small freely moving foreground objects with few and short feature tracks. We argue that all of these difficulties need to be handled online as structure-from-motion estimation progresses, and present an exemplary solution using the framework of probabilistic model-scoring.

[1]  Mei Han,et al.  Multiple Motion Scene Reconstruction from Uncalibrated Views , 2001, ICCV.

[2]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[3]  Yoshua Bengio,et al.  Pattern Recognition and Neural Networks , 1995 .

[4]  René Vidal,et al.  Motion Segmentation with Missing Data Using PowerFactorization and GPCA , 2004, CVPR.

[5]  Luc Van Gool,et al.  Reconstructing 3D trajectories of independently moving objects using generic constraints , 2004, Comput. Vis. Image Underst..

[6]  A. Shashua,et al.  On projection matrices P/sup k//spl rarr/P/sup 2/, k=3,...,6, and their applications in computer vision , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[7]  O. Faugeras,et al.  The Geometry of Multiple Images , 1999 .

[8]  Ruzena Bajcsy,et al.  Segmentation of range images as the search for geometric parametric models , 1995, International Journal of Computer Vision.

[9]  Rama Chellappa,et al.  Bayesian algorithms for simultaneous structure from motion estimation of multiple independently moving objects , 2005, IEEE Transactions on Image Processing.

[10]  David Suter,et al.  A Model-Selection Framework for Multibody Structure-and-Motion of Image Sequences , 2007, International Journal of Computer Vision.

[11]  René Vidal,et al.  Projective Factorization of Multiple Rigid-Body Motions , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Lior Wolf,et al.  On Projection Matrices and their Applications in Computer Vision , 2001, ICCV.

[13]  Kenichi Kanatani,et al.  Motion segmentation by subspace separation and model selection , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[14]  Reinhard Koch,et al.  Visual Modeling with a Hand-Held Camera , 2004, International Journal of Computer Vision.

[15]  Andrew W. Fitzgibbon,et al.  Multibody Structure and Motion: 3-D Reconstruction of Independently Moving Objects , 2000, ECCV.

[16]  S. Shankar Sastry,et al.  Two-View Multibody Structure from Motion , 2005, International Journal of Computer Vision.

[17]  Mei Han,et al.  Reconstruction of a Scene with Multiple Linearly Moving Objects , 2004, International Journal of Computer Vision.

[18]  Ali Azarbayejani,et al.  Segmentation of Rigidly Moving Objects Using Multiple Kalman Filters , 1994 .

[19]  Marc Pollefeys,et al.  A General Framework for Motion Segmentation: Independent, Articulated, Rigid, Non-rigid, Degenerate and Non-degenerate , 2006, ECCV.

[20]  Takeo Kanade,et al.  A multi-body factorization method for motion analysis , 1995, Proceedings of IEEE International Conference on Computer Vision.

[21]  James R. Bergen,et al.  Visual odometry , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[22]  Luc Van Gool,et al.  Reconstructing 3D independent motions using non-accidentalness , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[23]  René Vidal,et al.  A Benchmark for the Comparison of 3-D Motion Segmentation Algorithms , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  R. Vidal,et al.  Motion segmentation with missing data using PowerFactorization and GPCA , 2004, CVPR 2004.

[25]  Lior Wolf,et al.  On Projection Matrices $$\mathcal{P}^k \to \mathcal{P}^2 ,k = 3,...,6, $$ and their Applications in Computer Vision , 2004, International Journal of Computer Vision.

[26]  David Suter,et al.  Two-view multibody structure-and-motion with outliers through model selection , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Luc Van Gool,et al.  Background recognition in dynamic scenes with motion constraints , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[28]  Philip H. S. Torr,et al.  Bayesian Model Estimation and Selection for Epipolar Geometry and Generic Manifold Fitting , 2002, International Journal of Computer Vision.

[29]  David Suter,et al.  Robust Fitting by Adaptive-Scale Residual Consensus , 2004, ECCV.

[30]  Paul A. Beardsley,et al.  3D Model Acquisition from Extended Image Sequences , 1996, ECCV.

[31]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .