Appearance-Based Virtual-View Generation for Fly Through in a Real Dynamic Scene

We present appearance-based Virtual view generation which allows viewers to fly through a real dynamic scene. The scene is captured by synchronized multiple cameras. Arbitrary views are generated by interpolating two original camera-view images near the given viewpoint. The quality of the generated synthetic view is determined by the precision, consistency and density of correspondences between the two images. All or most of previous work that uses interpolation extracts the correspondences from these two images. However, not only is it difficult to do so reliably (the task requires a good stereo algorithm), but also the two images alone sometimes do not have enough information, due to problems such as occlusion. Instead, we take advantage of the fact that we have many views, from which we can extract much more reliable and comprehensive 3D geometry of the scene as a 3D model. The dense and precise correspondences between the two images, to be used for interpolation, are derived from this constructed 3D model. Our method of 3D modeling from multiple images uses the Multiple Baseline Stereo method and Shape from Silhouette method.

[1]  Lance Williams,et al.  View Interpolation for Image Synthesis , 1993, SIGGRAPH.

[2]  Takeo Kanade,et al.  The 3D Room: Digitizing Time-Varying 3D Events by Synchronized Multiple Video Streams , 1998 .

[3]  Takeo Kanade,et al.  Virtualized Reality: Constructing Virtual Worlds from Real Scenes , 1997, IEEE Multim..

[4]  Takeo Kanade,et al.  Constructing virtual worlds using dense stereo , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[5]  Steven M. Seitz,et al.  View morphing , 1996, SIGGRAPH.

[6]  Hideyuki Tamura,et al.  Viewpoint-dependent stereoscopic display using interpolation of multiviewpoint images , 1995, Electronic Imaging.

[7]  Michael Potmesil.  Generating octree models of 3D objects from their silhouettes in a sequence of images , 1987, Comput. Vis. Graph. Image Process..

[8]  Amnon Shashua,et al.  Novel View Synthesis by Cascading Trilinear Tensors , 1998, IEEE Trans. Vis. Comput. Graph..

[9]  Jake K. Aggarwal,et al.  Volume/surface octrees for the representation of three-dimensional objects , 1986, Comput. Vis. Graph. Image Process..

[10]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[11]  Katsushi Ikeuchi,et al.  Consensus surfaces for modeling 3D objects from multiple range images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[12]  Takeo Kanade,et al.  A Multiple-Baseline Stereo , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  O. Faugeras,et al.  3-D Reconstruction of Urban Scenes from Sequences of Images , 1995 .

[14]  Jake K. Aggarwal,et al.  Identification of 3D objects from multiple silhouettes using quadtrees/octrees , 1985, Comput. Vis. Graph. Image Process..

[15]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[16]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[17]  Hideo Saito,et al.  Modeling, Combining, and Rendering Dynamic Real-World Events From Image Sequences , 1998 .

[18]  Roger Y. Tsai,et al.  A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses , 1987, IEEE J. Robotics Autom..