VHS to VRML: 3D graphical models from video sequences

We describe a method to completely automatically recover 3D scene structure together with a camera for each frame from a sequence of images acquired by an unknown camera undergoing unknown movement. Previous approaches have used calibration objects or landmarks to recover this information, and are therefore often limited to a particular scale. The approach of this paper is far more general, since the "landmarks" are derived directly from the imaged scene texture. The method can be applied to a large class of scenes and motions, and is demonstrated for sequences of interior and exterior scenes using both controlled-motion and hand-held cameras. We demonstrate two applications of this technology. The first is the construction of 3D graphical models of the scene; the second is the insertion of virtual objects into the original image sequence. Other applications include image compression and frame interpolation.

[1]  Andrew W. Fitzgibbon,et al.  Automatic Camera Recovery for Closed or Open Image Sequences , 1998, ECCV.

[2]  Paul A. Beardsley,et al.  3D Model Acquisition from Extended Image Sequences , 1996, ECCV.

[3]  Philip H. S. Torr,et al.  Statistical detection of independent movement from a moving camera , 1993, Image Vis. Comput..

[4]  Shumin Zhai,et al.  Applications of augmented reality for human-robot communication , 1993, Proceedings of 1993 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS '93).

[5]  Andrew Zisserman,et al.  Automatic reconstruction of piecewise planar models from multiple views , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[6]  Cordelia Schmid,et al.  The Geometry and Matching of Curves in Multiple Views , 1998, ECCV.

[7]  Paul Debevec,et al.  Modeling and Rendering Architecture from Photographs , 1996, SIGGRAPH 1996.

[8]  Rajiv Gupta,et al.  Stereo from uncalibrated cameras , 1992, Proceedings 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[9]  H. C. Longuet-Higgins,et al.  A computer algorithm for reconstructing a scene from two projections , 1981, Nature.

[10]  Andrew Zisserman,et al.  Robust computation and parametrization of multiple view relations , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[11]  Soren W. Henriksen,et al.  Manual of photogrammetry , 1980 .

[12]  Reinhard Koch,et al.  Self-Calibration and Metric Reconstruction Inspite of Varying and Unknown Intrinsic Camera Parameters , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[13]  Andrew Zisserman,et al.  Wide baseline stereo matching , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[14]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[15]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[16]  Cordelia Schmid,et al.  Automatic line matching across views , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[17]  Rachid Deriche,et al.  A Robust Technique for Matching two Uncalibrated Images Through the Recovery of the Unknown Epipolar Geometry , 1995, Artif. Intell..

[18]  Philip H. S. Torr,et al.  Statistical detection of independent movement from a moving camera , 1993, Image Vis. Comput..

[19]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[20]  Amnon Shashua,et al.  Trilinearity in Visual Recognition by Alignment , 1994, ECCV.

[21]  Richard I. Hartley,et al.  A linear method for reconstruction from lines and points , 1995, Proceedings of IEEE International Conference on Computer Vision.

[22]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[23]  Olivier D. Faugeras,et al.  What can be seen in three dimensions with an uncalibrated stereo rig , 1992, ECCV.

[24]  O. D. Faugeras,et al.  Camera Self-Calibration: Theory and Experiments , 1992, ECCV.

[25]  Andrew Zisserman,et al.  Robust parameterization and computation of the trifocal tensor , 1997, Image Vis. Comput..

[26]  KanadeTakeo,et al.  Shape and motion from image streams under orthography , 1992 .