A Survey of Image-based Relighting Techniques

JVRB, 4(2007), no. 7. - Image-based Relighting (IBRL) has recently attracted a lot of research interest for its ability to relight real objects or scenes, from novel illuminations captured in natural/synthetic environments. Complex lighting effects such as subsurface scattering, interreflection, shadowing, mesostructural self-occlusion, refraction and other relevant phenomena can be generated using IBRL. The main advantage of image-based graphics is that the rendering time is independent of scene complexity as the rendering is actually a process of manipulating image pixels, instead of simulating light transport. The goal of this paper is to provide a complete and systematic overview of the research in Imagebased Relighting. We observe that essentially all IBRL techniques can be broadly classified into three categories (Fig. 9), based on how the scene/illumination information is captured: Reflectance function-based, Basis function-based and Plenoptic function-based. We discuss the characteristics of each of these categories and their representative methods. We also discuss about the sampling density and types of light source(s), relevant issues of IBRL.

[1]  Pieter Peers,et al.  Relighting with 4D incident light fields , 2003, ACM Trans. Graph..

[2]  Baining Guo,et al.  Synthesis of bidirectional texture functions on arbitrary surfaces , 2002, SIGGRAPH.

[3]  F. E. Nicodemus,et al.  Geometrical considerations and nomenclature for reflectance , 1977 .

[4]  Hans-Peter Seidel,et al.  DISCO: acquisition of translucent objects , 2004, ACM Trans. Graph..

[5]  Johan Wagemans,et al.  A perceptual quality scale for image-based relighting of glossy surfaces , 2005 .

[6]  Pat Hanrahan,et al.  All-frequency shadows using non-linear wavelet lighting approximation , 2003, ACM Trans. Graph..

[7]  Peter-Pike J. Sloan,et al.  Clustered principal components for precomputed radiance transfer , 2003, ACM Trans. Graph..

[8]  Tien-Tsin Wong,et al.  Image-based relighting: representation and compression , 2004 .

[9]  Rui Wang,et al.  All-frequency interactive relighting of translucent objects with single and multiple scattering , 2005, ACM Trans. Graph..

[10]  Steve Marschner,et al.  A practical model for subsurface light transport , 2001, SIGGRAPH.

[11]  Paul E. Debevec.  Image-Based Lighting , 2002, IEEE Computer Graphics and Applications.

[12]  Katsushi Ikeuchi,et al.  Appearance Based Object Modeling using Texture Database: Acquisition Compression and Rendering , 2002, Rendering Techniques.

[13]  Shree K. Nayar,et al.  Eyes for relighting , 2004, ACM Trans. Graph..

[14]  David J. Kriegman,et al.  Illumination cones for recognition under variable lighting: faces , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[15]  Andrew Gardner,et al.  Animatable Facial Reflectance Fields , 2004 .

[16]  Hans-Peter Seidel,et al.  Bayesian relighting , 2005, EGSR '05.

[17]  Ares Lagae,et al.  Interactive Rendering with Bidirectional Texture Functions , 2003, Comput. Graph. Forum.

[18]  Thomas Malzbender,et al.  Polynomial texture maps , 2001, SIGGRAPH.

[19]  Pieter Peers,et al.  Wavelet Environment matting , 2003, Rendering Techniques.

[20]  Athinodoros S. Georghiades,et al.  Recovering 3-D Shape and Reflectance From a Small Number of Photographs , 2003, Rendering Techniques.

[21]  Hans-Peter Seidel,et al.  Image-based reconstruction of spatial appearance and geometric detail , 2003, TOGS.

[22]  David J. Kriegman,et al.  Image-based modeling and rendering of surfaces with arbitrary BRDFs , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[23]  Tsuhan Chen,et al.  Spectral analysis for sampling image-based rendering data , 2003, IEEE Trans. Circuits Syst. Video Technol..

[24]  Paul E. Debevec.  Pursuing Reality with Image-Based Modeling, Rendering, and Lighting , 2000, SMILE.

[25]  Harry Shum,et al.  Relighting with the Reflected Irradiance Field: Representation, Sampling and Reconstruction , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[26]  Julie Dorsey,et al.  Effic ient Re-rendering of Naturally Illuminated Environments , 1994 .

[27]  Tsuhan Chen,et al.  A survey on image-based rendering - representation, sampling and compression , 2004, Signal Process. Image Commun..

[28]  André Gagalowicz,et al.  Image-based rendering of diffuse, specular and glossy surfaces from a single image , 2001, SIGGRAPH.

[29]  Chi-Wing Fu,et al.  Interactive Relighting of Panoramas , 2001, IEEE Computer Graphics and Applications.

[30]  Kazufumi Kaneda,et al.  A Quick Rendering Method Using Basis Functions for Interactive Lighting Design , 1995, Comput. Graph. Forum.

[31]  David Salesin,et al.  Environment matting and compositing , 1999, SIGGRAPH.

[32]  Daniel Keren,et al.  Image detection under varying illumination and pose , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[33]  Paul E. Debevec,et al.  Unlighting the Parthenon , 2004, SIGGRAPH '04.

[34]  E. Adelson,et al.  The Plenoptic Function and the Elements of Early Vision , 1991 .

[35]  Pieter Peers,et al.  Smooth Reconstruction and Compact Representation of Reflectance Functions for Image-based Relighting , 2004, Rendering Techniques.

[36]  Jan Kautz,et al.  Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments , 2002 .

[37]  Tien-Tsin Wong,et al.  Image-based Rendering with Controllable Illumination , 1997, Rendering Techniques.

[38]  Demetri Terzopoulos,et al.  TensorTextures: multilinear image-based rendering , 2004, ACM Trans. Graph..

[39]  David Salesin,et al.  Environment matting extensions: towards higher accuracy and real-time capture , 2000, SIGGRAPH.

[40]  Stephen Lin,et al.  Single-image reflectance estimation for relighting by iterative soft grouping , 2002, 10th Pacific Conference on Computer Graphics and Applications, 2002. Proceedings..

[41]  S. Marschner,et al.  Inverse Rendering for Computer Graphics , 1998 .

[42]  Pat Hanrahan,et al.  An efficient representation for irradiance environment maps , 2001, SIGGRAPH.

[43]  Wojciech Matusik,et al.  Acquisition and Rendering of Transparent and Refractive Objects , 2002, Rendering Techniques.

[44]  Andrew Gardner,et al.  Performance relighting and reflectance transformation with time-multiplexed illumination , 2005, ACM Trans. Graph..

[45]  Paul E. Debevec,et al.  Acquiring the reflectance field of a human face , 2000, SIGGRAPH.

[46]  Harry Shum,et al.  Review of image-based rendering techniques , 2000, Visual Communications and Image Processing.

[47]  Tsuhan Chen,et al.  A statistical framework for image-based relighting , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[48]  Wojciech Matusik,et al.  Progressively-Refined Reflectance Functions from Natural Illumination , 2004 .

[49]  Paul Debevec.  Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography , 2008, SIGGRAPH Classes.

[50]  Jan Kautz,et al.  Fast Arbitrary BRDF Shading for Low-Frequency Lighting Using Spherical Harmonics , 2002, Rendering Techniques.

[51]  David J. Kriegman,et al.  From Few to Many: Illumination Cone Models for Face Recognition under Variable Lighting and Pose , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[52]  James T. Kajiya,et al.  Anisotropic reflection models , 1985, SIGGRAPH.

[53]  Harry Shum,et al.  Plenoptic sampling , 2000, SIGGRAPH.

[54]  Pat Hanrahan,et al.  Triple product wavelet integrals for all-frequency relighting , 2004, ACM Trans. Graph..