Hardware‐Accelerated Rendering of Photo Hulls

This paper presents an efficient hardware‐accelerated method for novel view synthesis from a set of images or videos. Our method is based on the photo hull representation, which is the maximal photo‐consistent shape. We avoid the explicit reconstruction of photo hulls by adopting a view‐dependent plane‐sweeping strategy. From the target viewpoint slicing planes are rendered with reference views projected onto them. Graphics hardware is exploited to verify the photo‐consistency of each rasterized fragment. Visibilities with respect to reference views are properly modeled, and only photo‐consistent fragments are kept and colored in the target view. We present experiments with real images and animation sequences. Thanks to the more accurate shape of the photo hull representation, our method generates more realistic rendering results than methods based on visual hulls. Currently, we achieve rendering frame rates of 2–3 fps. Compared to a pure software implementation, the performance of our hardware‐accelerated method is approximately 7 times faster.

[1]  Benjamin Lok,et al.  Online model reconstruction for interactive virtual environments , 2001, I3D '01.

[2]  Richard Szeliski,et al.  Rapid octree construction from image sequences , 1993 .

[3]  Darius Burschka,et al.  Advances in Computational Stereo , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[4]  Ramesh C. Jain,et al.  Reality modeling and visualization from multiple video sequences , 1996, IEEE Computer Graphics and Applications.

[5]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[6]  Bruce G. Baumgart,et al.  Geometric modeling for computer vision. , 1974 .

[7]  C. Dyer.  Volumetric Scene Reconstruction from Multiple Views , 2001 .

[8]  Ruigang Yang,et al.  Real‐Time Consensus‐Based Scene Reconstruction Using Commodity Graphics Hardware † , 2003, Comput. Graph. Forum.

[9]  Peter Eisert,et al.  Multi-hypothesis, volumetric reconstruction of 3-D objects from multiple calibrated camera views , 1999, 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258).

[10]  Thomas Malzbender,et al.  Generalized Voxel Coloring , 1999, Workshop on Vision Algorithms.

[11]  Pierre Poulin,et al.  Interactive Point-based Modeling of Complex Objects from Images , 2003, Graphics Interface.

[12]  Wojciech Matusik,et al.  Polyhedral Visual Hulls for Real-Time Rendering , 2001, Rendering Techniques.

[13]  Steven M. Seitz,et al.  Photorealistic Scene Reconstruction by Voxel Coloring , 1997, International Journal of Computer Vision.

[14]  Ronald W. Schafer,et al.  Image-based photo hulls for fast and photo-realistic new view synthesis , 2003, Real Time Imaging.

[15]  Ramesh Raskar,et al.  Image-based visual hulls , 2000, SIGGRAPH.

[16]  Ruigang Yang,et al.  Multi-resolution real-time stereo on commodity graphics hardware , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[17]  Nader Bagherzadeh,et al.  Hardware Accelerated Voxel Carving , 2002 .

[18]  Paul Debevec,et al.  Modeling and Rendering Architecture from Photographs , 1996, SIGGRAPH 1996.

[19]  Takashi Matsuyama,et al.  Generation, visualization, and editing of 3D video , 2002, Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission.

[20]  Hans-Peter Seidel,et al.  Hardware-Accelerated Visual Hull Reconstruction and Rendering , 2003, Graphics Interface.

[21]  Michael Potmesil.  Generating octree models of 3D objects from their silhouettes in a sequence of images , 1987, Comput. Vis. Graph. Image Process..

[22]  A. Laurentini,et al.  The Visual Hull Concept for Silhouette-Based Image Understanding , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  Kiriakos N. Kutulakos,et al.  A Theory of Shape by Space Carving , 2000, International Journal of Computer Vision.