Interactive Virtual Relighting and Remodeling of Real Scenes

Lighting design is often tedious due to the required physical manipulation of real light sources and objects. As an alternative, we present an interactive system to virtually modify the lighting and geometry of scenes with both real and synthetic objects, including mixed real/virtual lighting and shadows. In our method, real scene geometry is first approximately reconstructed from photographs. Additional images are taken from a single viewpoint with a real light in different positions to estimate reflectance. A filtering process is used to compensate for inaccuracies, and per image reflectances are averaged to generate an approximate reflectance image for the given viewpoint, removing shadows in the process. This estimate is used to initialise a global illumination hierarchical radiosity system, representing real-world secondary illumination; the system is optimized for interactive updates. Direct illumination from lights is calculated separately using ray-casting and a table for efficient reuse of data where appropriate. Our system allows interactive modification of light emission and object positions, all with mixed real/virtual illumination effects. Real objects can also be virtually removed using texture-filling algorithms for reflectance estimation.

[1]  Donald P. Greenberg,et al.  Improved Computational Methods for Ray Tracing , 1984, TOGS.

[2]  James R. Bergen,et al.  Pyramid-based texture analysis/synthesis , 1995, Proceedings., International Conference on Image Processing.

[3]  Katsushi Ikeuchi,et al.  Object shape and reflectance modeling from observation , 1997, SIGGRAPH.

[4]  Jitendra Malik,et al.  Recovering photometric properties of architectural scenes from photographs , 1998, SIGGRAPH.

[5]  Pierre Poulin,et al.  Interactively Modeling with Photogrammetry , 1998, Rendering Techniques.

[6]  A. Fournier,et al.  Common Illumination between Real and Computer Generated Scenes , 1992 .

[7]  O. Faugeras,et al.  3-D Reconstruction of Urban Scenes from Sequences of Images , 1995 .

[8]  Bob Francis,et al.  Silicon Graphics Inc. , 1993 .

[9]  Tomoyuki Nishita,et al.  A montage method: the overlaying of the computer generated images onto a background photograph , 1986, SIGGRAPH.

[10]  Thomas F. El-maraghi.  An Implementation of Heeger and Bergen's Texture Analysis/Synthesis Algorithm , 1997 .

[11]  John R. Wallace,et al.  Shaft Culling for Efficient Ray-Cast Radiosity , 1994 .

[12]  Erin Shaw.  Hierarchical Radiosity for Dynamic Environments , 1997, Comput. Graph. Forum.

[13]  François X. Sillion,et al.  A Unified Hierarchical Algorithm for Global Illumination with Scattering Volumes and Object Clusters , 1995, IEEE Trans. Vis. Comput. Graph..

[14]  Lucas Pereira,et al.  Image replacement through texture synthesis , 1997, Proceedings of International Conference on Image Processing.

[15]  F. W. Jansen,et al.  Photorealistic rendering in computer graphics : proceedings of the Second Eurographics Workshop on Rendering , 1994 .

[16]  Paul Debevec,et al.  Inverse global illumination: Recovering re?ectance models of real scenes from photographs , 1998 .

[17]  George Drettakis,et al.  Interactive Common Illumination for Computer Augmented Reality , 1997, Rendering Techniques.

[18]  Gregory J. Ward,et al.  The RADIANCE lighting simulation and rendering system , 1994, SIGGRAPH.

[19]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[20]  George Drettakis,et al.  Interactive update of global illumination using a line-space hierarchy , 1997, SIGGRAPH.

[21]  Jan J. Koenderink,et al.  REALISE: reconstruction of REALity from Image SEquences , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[22]  Pat Hanrahan,et al.  A rapid hierarchical radiosity algorithm , 1991, SIGGRAPH.

[23]  S. P. Mudur,et al.  Three-dimensional computer vision: a geometric viewpoint , 1993 .

[24]  George Drettakis,et al.  Interactive modification of real and virtual lights for augmented reality , 1998, SIGGRAPH '98.

[25]  Daniel R. Baum,et al.  Improving radiosity solutions through the use of analytically determined form-factors , 1989, SIGGRAPH.