A Multibody Factorization Method for Independently Moving Objects

The structure-from-motion problem has been extensively studied in the field of computer vision. Yet, the bulk of the existing work assumes that the scene contains only a single moving object. The more realistic case where an unknown number of objects move in the scene has received little attention, especially for its theoretical treatment. In this paper we present a new method for separating and recovering the motion and shape of multiple independently moving objects in a sequence of images. The method does not require prior knowledge of the number of objects, nor is dependent on any grouping of features into an object at the image level. For this purpose, we introduce a mathematical construct of object shapes, called the shape interaction matrix, which is invariant to both the object motions and the selection of coordinate systems. This invariant structure is computable solely from the observed trajectories of image features without grouping them into individual objects. Once the matrix is computed, it allows for segmenting features into objects by the process of transforming it into a canonical form, as well as recovering the shape and motion of each object. The theory works under a broad set of projection models (scaled orthography, paraperspective and affine) but they must be linear, so it excludes projective “cameras”.

[1]  THE OPTICAL SOCIETY OF AMERICA , 1923 .

[2]  Thomas L. Marzetta,et al.  Detection, Estimation, and Modulation Theory , 1976 .

[3]  Ralph Otto Schmidt,et al.  A signal subspace approach to multiple emitter location and spectral estimation , 1981 .

[4]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[5]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[6]  Andrew B. Watson,et al.  A look at motion in the frequency domain , 1983 .

[7]  Ellen C. Hildreth,et al.  Measurement of Visual Motion , 1984 .

[8]  S Ullman,et al.  Maximizing Rigidity: The Incremental Recovery of 3-D Structure from Rigid and Nonrigid Motion , 1984, Perception.

[9]  E H Adelson,et al.  Spatiotemporal energy models for the perception of motion. , 1985, Journal of the Optical Society of America. A, Optics and image science.

[10]  Gilad Adiv,et al.  Determining Three-Dimensional Motion and Structure from Optical Flow Generated by Several Moving Objects , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Yoshiaki Shirai,et al.  Three-Dimensional Computer Vision , 1987, Symbolic Computation.

[12]  G. Stewart,et al.  A Generalization of the Eckart-Young-Mirsky , 1987 .

[13]  G. Stewart,et al.  A generalization of the Eckart-Young-Mirsky matrix approximation theorem , 1987 .

[14]  D J Heeger,et al.  Model for the extraction of image flow. , 1987, Journal of the Optical Society of America. A, Optics and image science.

[15]  J. Demmel.  The smallest perturbation of a submatrix which lowers the rank and constrained total least squares problems , 1987 .

[16]  Shmuel Peleg,et al.  Computing two motions from three frames , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[17]  Takeo Kanade,et al.  Shape and motion without depth , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[18]  T. Boult,et al.  Factorization-based segmentation of motions , 1991, Proceedings of the IEEE Workshop on Visual Motion.

[19]  J J Koenderink,et al.  Affine structure from motion. , 1991, Journal of the Optical Society of America. A, Optics and image science.

[20]  P. Burt,et al.  Mechanisms for isolating component patterns in the sequential analysis of multiple motion , 1991, Proceedings of the IEEE Workshop on Visual Motion.

[21]  Radu S. Jasinschi.  Intrinsic Constraints in Space-Time Filtering: A New Approach to Representing Uncertainty in Low-Level Vision , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[22]  P. Anandan,et al.  Hierarchical Model-Based Motion Estimation , 1992, ECCV.

[23]  G. Stewart.  Determining rank in the presence of error , 1992 .

[24]  Nassir Navab,et al.  From Multiple Objects Motion Analysis To Behavior-Based Object Recognition , 1992, ECAI.

[25]  A. Rosenfeld,et al.  Perceptual motion transparency : the role of geometrical information , 1992 .

[26]  R. S. Jasinchi.  Intrinsic constraints in space-time filtering: a new approach to representing uncertainty in low-level vision , 1992 .

[27]  D. Sinclair.  Motion segmentation and local structure , 1993, 1993 (4th) International Conference on Computer Vision.

[28]  C Tomasi,et al.  Shape and motion from image streams: a factorization method. , 1992, Proceedings of the National Academy of Sciences of the United States of America.

[29]  G. W. Stewart,et al.  On the Early History of the Singular Value Decomposition , 1993, SIAM Rev..

[30]  O. Faugeras.  Three-dimensional computer vision: a geometric viewpoint , 1993 .

[31]  Reg G. Willson.  Modeling and calibration of automated zoom lenses , 1994, Other Conferences.

[32]  Yung-Sheng Chen,et al.  Automatic approach to mapping a lifelike 2.5D human face , 1994, Image Vis. Comput..

[33]  C. W. Gear,et al.  Feature grouping in moving objects , 1994, Proceedings of 1994 IEEE Workshop on Motion of Non-rigid and Articulated Objects.

[34]  Takeo Kanade,et al.  A sequential factorization method for recovering shape and motion from image streams , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[35]  Takeo Kanade,et al.  A Paraperspective Factorization Method for Shape and Motion Recovery , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  David J. Heeger,et al.  Optical flow using spatiotemporal filters , 2004, International Journal of Computer Vision.

[37]  Takeo Kanade,et al.  Shape and motion from image streams under orthography: a factorization method , 1992, International Journal of Computer Vision.

[38]  Michal Irani,et al.  Computing occluding and transparent motions , 1994, International Journal of Computer Vision.

[39]  C. Mellon,et al.  A Fast Feature Tracker for Image Sequence Analysis , 2006 .