Space-Time-Scale Registration of Dynamic Scene Reconstructions

The paper presents a method for multi-dimensional registration of two video streams. The sequences are captured by two hand-held cameras moving independently with respect to each other, both observing one object rigidly moving apart from the background. The method is based on uncalibrated Structure-from-Motion (SfM) to extract 3D models for the foreground object and the background, as well as for their relative motion. It fixes the relative scales between the scene parts within and between the videos. It also provides the registration between all partial 3D models, and the temporal synchronization between the videos. The crux is that not a single point on the foreground or background needs to be in common between both video streams. Extensions to more than two cameras and multiple foreground objects are possible.

[1]  S. Shankar Sastry,et al.  Two-View Segmentation of Dynamic Scenes from the Multibody Fundamental Matrix , 2002 .

[2]  Jitendra Malik,et al.  Motion segmentation and tracking using normalized cuts , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[3]  P. Torr.  Geometric motion segmentation and model selection , 1998, Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[4]  T. Kanade,et al.  A multi-body factorization method for motion analysis , 1995, ICCV 1995.

[5]  Roger Y. Tsai,et al.  A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses , 1987, IEEE J. Robotics Autom..

[6]  Lior Wolf,et al.  Sequence-to-Sequence Self Calibration , 2002, ECCV.

[7]  Amnon Shashua,et al.  Trajectory Triangulation: 3D Reconstruction of Moving Points from a Monocular Image Sequence , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Yaron Caspi,et al.  Alignment of non-overlapping sequences , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[9]  Luc Van Gool,et al.  Reconstructing 3D independent motions using non-accidentalness , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[10]  A. Shashua,et al.  On projection matrices P/sup k//spl rarr/P/sup 2/, k=3,...,6, and their applications in computer vision , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[11]  Matthew Brand,et al.  Morphable 3D models from video , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[12]  Andrew W. Fitzgibbon,et al.  Multibody Structure and Motion: 3-D Reconstruction of Independently Moving Objects , 2000, ECCV.

[13]  René Vidal.  A Factorization Method for 3 D Multi-body Motion Estimation and Segmentation ∗ , 2002 .

[14]  Yiu Cheung Shiu,et al.  Comments on 'Calibration of wrist-mounted robotic sensors by solving homogeneous transform equations of the form AX=XB' [with reply] , 1991, IEEE Trans. Robotics Autom..

[15]  Luc Van Gool,et al.  Reconstructing 3D trajectories of independently moving objects using generic constraints , 2004, Comput. Vis. Image Underst..

[16]  Henning Biermann,et al.  Recovering non-rigid 3D shape from image streams , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[17]  M. Hebert,et al.  The Representation, Recognition, and Locating of 3-D Objects , 1986 .

[18]  Marc Pollefeys,et al.  Synchronization and calibration of camera networks from silhouettes , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[19]  Yiu Cheung Shiu,et al.  Calibration of wrist-mounted robotic sensors by solving homogeneous transform equations of the form AX=XB , 1989, IEEE Trans. Robotics Autom..

[20]  Fadi Dornaika,et al.  Hand-Eye Calibration , 1995, Int. J. Robotics Res..

[21]  Christopher M. Bishop,et al.  Non-linear Bayesian Image Modelling , 2000, ECCV.

[22]  Mads Nielsen,et al.  Computer Vision — ECCV 2002 , 2002, Lecture Notes in Computer Science.

[23]  Luc Van Gool,et al.  Background recognition in dynamic scenes with motion constraints , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[24]  Marc Pollefeys,et al.  Multiple view geometry , 2005 .

[25]  Lior Wolf,et al.  Correspondence-free Synchronization and Reconstruction in a Non-rigid Scene , 2004 .

[26]  M. Irani,et al.  Spatio-Temporal Alignment of Sequences , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[27]  M. Irani,et al.  Multi-body Segmentation : Revisiting Motion Consistency , 2002 .

[28]  Lior Wolf,et al.  On Projection Matrices and their Applications in Computer Vision , 2001, ICCV.