Image-based photo hulls for fast and photo-realistic new view synthesis

We present an efficient image-based rendering algorithm that generates views of a scene's photo hull. The photo hull is the largest 3D shape that is photo-consistent with photographs taken of the scene from multiple viewpoints. Our algorithm, image-based photo hulls (IBPH), like the image-based visual hulls (IBVH) algorithm from Matusik et al. on which it is based, takes advantage of epipolar geometry to efficiently reconstruct the geometry and visibility of a scene. Our IBPH algorithm differs from IBVH in that it utilizes the color information of the images to identify scene geometry. These additional color constraints result in more accurately reconstructed geometry, which often projects to better synthesized virtual views of the scene. We demonstrate our algorithm running in a realtime 3D telepresence application using video data acquired from multiple viewpoints.

[1]  C. Dyer.  Volumetric Scene Reconstruction from Multiple Views , 2001 .

[2]  Takeo Kanade,et al.  Shape and motion carving in 6D , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[3]  Ronald W. Schafer,et al.  Image-based photo hulls , 2002, Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission.

[4]  Steven M. Seitz,et al.  Photorealistic Scene Reconstruction by Voxel Coloring , 1997, International Journal of Computer Vision.

[5]  Olivier D. Faugeras,et al.  Variational principles, surface evolution, PDEs, level set methods, and the stereo problem , 1998, IEEE Trans. Image Process..

[6]  Peter Eisert,et al.  Multi-hypothesis, volumetric reconstruction of 3-D objects from multiple calibrated camera views , 1999, 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258).

[7]  Thomas Malzbender,et al.  Generalized Voxel Coloring , 1999, Workshop on Vision Algorithms.

[8]  Peter Eisert,et al.  3-D reconstruction of real-world objects using extended voxels , 2000, Proceedings 2000 International Conference on Image Processing (Cat. No.00CH37101).

[9]  Richard Szeliski,et al.  Layered depth images , 1998, SIGGRAPH.

[10]  Roberto Cipolla,et al.  A Probabilistic Framework for Space Carving , 2001, ICCV.

[11]  Saied Moezzi,et al.  Virtual View Generation for 3D Digital Video , 1997, IEEE Multim..

[12]  Ramesh Raskar,et al.  Image-based visual hulls , 2000, SIGGRAPH.

[13]  Larry S. Davis,et al.  Foundations of Image Understanding , 2001 .

[14]  O. Faugeras,et al.  Variational principles, surface evolution, PDE's, level set methods and the stereo problem , 1998, 5th IEEE EMBS International Summer School on Biomedical Imaging, 2002..

[15]  Richard Szeliski,et al.  Rapid octree construction from image sequences , 1993 .

[16]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[17]  A. Laurentini,et al.  The Visual Hull Concept for Silhouette-Based Image Understanding , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  Kiriakos N. Kutulakos,et al.  A Theory of Shape by Space Carving , 2000, International Journal of Computer Vision.

[19]  Kiriakos N. Kutulakos,et al.  Multi-View Scene Capture by Surfel Sampling: From Video Streams to Non-Rigid 3D Motion, Shape and Reflectance , 2002, International Journal of Computer Vision.

[20]  Takeo Kanade,et al.  Constructing virtual worlds using dense stereo , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).