Appearance-based virtual view generation from multicamera videos captured in the 3-D room

We present an appearance-based virtual view generation method that allows viewers to fly through a real dynamic scene. The scene is captured by multiple synchronized cameras. Arbitrary views are generated by interpolating two original camera-views near the given viewpoint. The quality of the generated synthetic view is determined by the precision, consistency and density of correspondences between the two images. All or most of previous work that uses interpolation extracts the correspondences from these two images. However, not only is it difficult to do so reliably (the task requires a good stereo algorithm), but also the two images alone sometimes do not have enough information, due to problems such as occlusion. Instead, we take advantage of the fact that we have many views, from which we can extract much more reliable and comprehensive 3D geometry of the scene as a 3D model. Dense and precise correspondences between the two images, to be used for interpolation, are obtained using this constructed 3D model.

[1]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[2]  Václav Hlavác,et al.  Rendering real-world objects using view interpolation , 1995, Proceedings of IEEE International Conference on Computer Vision.

[3]  Yizhou Yu,et al.  Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping , 1998, Rendering Techniques.

[4]  William E. Lorensen,et al.  Marching cubes: a high resolution 3D surface construction algorithm , 1996 .

[5]  Yiannis Aloimonos,et al.  Spatio-Temporal Stereo Using Multi-Resolution Subdivision Surfaces , 2004, International Journal of Computer Vision.

[6]  Takeo Kanade,et al.  Shape and motion from image streams under orthography: a factorization method , 1992, International Journal of Computer Vision.

[7]  Katsushi Ikeuchi,et al.  Consensus surfaces for modeling 3D objects from multiple range images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[8]  Paul Debevec,et al.  Modeling and Rendering Architecture from Photographs , 1996, SIGGRAPH 1996.

[9]  Ramesh Raskar,et al.  Image-based visual hulls , 2000, SIGGRAPH.

[10]  Adrian Hilton,et al.  Reliable Surface Reconstructiuon from Multiple Range Images , 1996, ECCV.

[11]  Reinhard Koch,et al.  Self-calibration and metric reconstruction in spite of varying and unknown internal camera parameters , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[12]  Roger Y. Tsai,et al.  A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses , 1987, IEEE J. Robotics Autom..

[13]  Lance Williams,et al.  View Interpolation for Image Synthesis , 1993, SIGGRAPH.

[14]  Ronald P. Uhlig,et al.  The office of the future , 1979 .

[15]  Takeo Kanade,et al.  Three-dimensional scene flow , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[16]  Larry S. Davis,et al.  3-D model-based tracking of humans in action: a multi-view approach , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[17]  Michael Garland,et al.  Optimal triangulation and quadric-based surface simplification , 1999, Comput. Geom..

[18]  Michael Potmesil.  Generating octree models of 3D objects from their silhouettes in a sequence of images , 1987, Comput. Vis. Graph. Image Process..

[19]  Steven M. Seitz,et al.  Photorealistic Scene Reconstruction by Voxel Coloring , 1997, International Journal of Computer Vision.

[20]  O. Faugeras,et al.  3-D Reconstruction of Urban Scenes from Sequences of Images , 1995 .

[21]  Steven M. Seitz,et al.  View morphing , 1996, SIGGRAPH.

[22]  Jake K. Aggarwal,et al.  Identification of 3D objects from multiple silhouettes using quadtrees/octrees , 1985, Comput. Vis. Graph. Image Process..

[23]  Takeo Kanade,et al.  The 3D Room: Digitizing Time-Varying 3D Events by Synchronized Multiple Video Streams , 1998 .

[24]  Takeo Kanade,et al.  Constructing virtual worlds using dense stereo , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[25]  Greg Welch,et al.  The office of the future: a unified approach to image-based modeling and spatially immersive displays , 1998, SIGGRAPH.

[26]  Reinhard Koch,et al.  Self-Calibration and Metric Reconstruction Inspite of Varying and Unknown Intrinsic Camera Parameters , 1999, International Journal of Computer Vision.

[27]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[28]  Hideyuki Tamura,et al.  Viewpoint-dependent stereoscopic display using interpolation of multiviewpoint images , 1995, Electronic Imaging.

[29]  Thaddeus Beier,et al.  Feature-based image metamorphosis , 1992, SIGGRAPH.

[30]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[31]  Takeo Kanade,et al.  Virtualized Reality: Constructing Virtual Worlds from Real Scenes , 1997, IEEE Multim..

[32]  Hideyuki Tamura,et al.  Mixed Reality: Future Dreams Seen at the Border between Real and Virtual Worlds , 2001, IEEE Computer Graphics and Applications.

[33]  Saied Moezzi,et al.  Virtual View Generation for 3D Digital Video , 1997, IEEE Multim..

[34]  Marc Levoy,et al.  A volumetric method for building complex models from range images , 1996, SIGGRAPH.

[35]  Ramesh C. Jain,et al.  Multiple perspective interactive video , 1995, Proceedings of the International Conference on Multimedia Computing and Systems.

[36]  Amnon Shashua,et al.  Novel View Synthesis by Cascading Trilinear Tensors , 1998, IEEE Trans. Vis. Comput. Graph..

[37]  Takeo Kanade,et al.  A multiple-baseline stereo , 1991, Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[38]  Hideo Saito,et al.  Modeling, Combining, and Rendering Dynamic Real-World Events From Image Sequences , 1998 .