Interactive In-Shader Image-Based Visual Hull Reconstruction and Compositing of Actors in a Distributed Ray Tracing Framework

High-quality mixed reality rendering is the art of combining synthetic and photographic images in a photorealistically consistent way. Most rendering methods used in this area are based on ray tracing algorithms and take hours of computing time. With interactive ray tracing having become available in recent years it is obvious to explore its potential for mixed reality applications, like virtual TV studios featuring photorealistic live insertion of human actors in rendered background scenes. In this paper we present a method to seamlessly integrate live actor performance into the distributed interactive ray tracing framework OpenRT. An image based visual hull shader is used to create a 3D representation of the actor. This allows for the full spectrum of ray tracing based effects like reflection, refraction, and shadows in the composite image.

[1]  Philipp Slusallek,et al.  Realtime Ray Tracing and its use for Interactive Global Illumination , 2003, Eurographics.

[2]  Edmond Boyer,et al.  Real-Time Capture, Reconstruction and Insertion into Virtual World of Human Actors , 2003, VVG.

[3]  Takeo Kanade,et al.  A real time system for robust 3D voxel reconstruction of human motions , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[4]  François X. Sillion,et al.  A Real-Time System for Full Body Interaction with Virtual Worlds , 2004, EGVE.

[5]  Geoff Wyvill,et al.  Analysis of an algorithm for fast ray tracing using uniform space subdivision , 1988, The Visual Computer.

[6]  Philipp Slusallek,et al.  The OpenRT Application Programming Interface - Towards A Common API for Interactive Ray Tracing , 2003 .

[7]  Philipp Slusallek,et al.  Sttreaming Video Textures for Mixed Reality Applications in Interactive Ray Tracing Environments , 2003, VMV.

[8]  James F. Blinn,et al.  Blue screen matting , 1996, SIGGRAPH.

[9]  Takeshi Shakunaga,et al.  Color blending based on viewpoint and surface normal for generating images from any viewpoint using multiple cameras , 2003, Proceedings of IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, MFI2003..

[10]  Hans-Peter Seidel,et al.  Free-viewpoint video of human actors , 2003, ACM Trans. Graph..

[11]  Wojciech Matusik,et al.  Polyhedral Visual Hulls for Real-Time Rendering , 2001, Rendering Techniques.

[12]  Hans-Peter Seidel,et al.  Hardware-Accelerated Visual Hull Reconstruction and Rendering , 2003, Graphics Interface.

[13]  Yizhou Yu,et al.  Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping , 1998, Rendering Techniques.

[14]  Armin Gruen,et al.  Videometrics and Optical Methods for 3d Shape Measurement , 2000 .

[15]  Thomas Malzbender,et al.  A Survey of Methods for Volumetric Scene Reconstruction from Photographs , 2001, VG.

[16]  Hans-Peter Seidel,et al.  A Hybrid Hardware-Accelerated Algorithm for High Quality Rendering of Visual Hulls , 2004, Graphics Interface.

[17]  Philipp Slusallek,et al.  State of the Art in Interactive Ray Tracing , 2001, Eurographics.

[18]  Ramesh Raskar,et al.  Image-based visual hulls , 2000, SIGGRAPH.

[19]  Philipp Slusallek,et al.  A Scalable Approach to Interactive Global Illumination , 2003, Comput. Graph. Forum.

[20]  A. Laurentini,et al.  The Visual Hull Concept for Silhouette-Based Image Understanding , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Oliver Grau,et al.  Use of 3D techniques for virtual production , 2000, IS&T/SPIE Electronic Imaging.