Image-based relighting: representation and compression

Image-based modeling and rendering has been demonstrated as a cost-effective and efficient approach to computer games and virtual reality applications. The computational model that most image-based techniques are based on is the plenoptic function. Since the original formulation of the plenoptic function does not include illumination, previous image-based applications simply assume that the illumination is fixed. We have proposed a new formulation of the plenoptic function, called the plenoptic illumination function, which explicitly specifies the illumination component. Techniques based on this formulation can be extended to support relighting as well as view interpolation. The core of this framework is compression, and we show how to exploit three types of data correlation, the intra-pixel, the inter-pixel and the inter-channel correlations, in order to achieve a manageable storage size. The proposed coding method outperforms JPEG, JPEG2000 and MPEG.

[1]  R. Courant,et al.  Methods of Mathematical Physics , 1962 .

[2]  Harry Shum,et al.  Relighting with the Reflected Irradiance Field: Representation, Sampling and Reconstruction , 2004, International Journal of Computer Vision.

[3]  R. Gray.  Source Coding Theory , 1989 .

[4]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[5]  Tien-Tsin Wong,et al.  Image-based Rendering with Controllable Illumination , 1997, Rendering Techniques.

[6]  Tien-Tsin Wong.  Image-based relighting as the sampling and reconstruction of the plenoptic illumination function , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[7]  P. Yip,et al.  Discrete Cosine Transform: Algorithms, Advantages, Applications , 1990 .

[8]  Jerry D. Gibson,et al.  Distributions of the Two-Dimensional DCT Coefficients for Images , 1983, IEEE Trans. Commun..

[9]  Dani Lischinski,et al.  Discontinuity meshing for accurate radiosity , 1992, IEEE Computer Graphics and Applications.

[10]  M. Charrier,et al.  JPEG2000, the next millennium compression standard for still images , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[11]  E. Adelson,et al.  The Plenoptic Function and the Elements of Early Vision , 1991 .

[12]  Michael W. Marcellin,et al.  Comparison of different methods of classification in subband coding of images , 1997, IEEE Trans. Image Process..

[13]  Andrew Chi-Sing Leung,et al.  Compression of illumination-adjustable images , 2003, IEEE Trans. Circuits Syst. Video Technol..

[14]  Hiroshi Murase,et al.  Dimensionality of illumination in appearance matching , 1996, Proceedings of IEEE International Conference on Robotics and Automation.

[15]  Julie Dorsey,et al.  Effic ient Re-rendering of Naturally Illuminated Environments , 1994 .

[16]  Hiroshi Murase,et al.  Dimensionality of Illumination Manifolds in Appearance Matching , 1996, Object Representation in Computer Vision.

[17]  Andrew Chi-Sing Leung,et al.  Data compression for a massive image data set in IBMR , 2003, Proceedings Computer Graphics International 2003.

[18]  Peter Schröder,et al.  Spherical wavelets: efficiently representing functions on the sphere , 1995, SIGGRAPH.

[19]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[20]  Nelson L. Max,et al.  Bidirectional reflection functions from surface bump maps , 1987, SIGGRAPH.

[21]  Leonard McMillan,et al.  Plenoptic Modeling: An Image-Based Rendering System , 2023 .

[22]  Maryline Charrier.  JPEG2000: A New Standard for Still Image Compression , 1999 .

[23]  Harry Shum,et al.  Rendering with concentric mosaics , 1999, SIGGRAPH.

[24]  Stephen H. Westin,et al.  A global illumination solution for general reflectance distributions , 1991, SIGGRAPH.

[25]  Harry Shum,et al.  Plenoptic sampling , 2000, SIGGRAPH.

[26]  David J. Kriegman,et al.  What is the set of images of an object under all possible lighting conditions? , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[27]  Chi-Wing Fu,et al.  Real-time relighting of compressed panoramas , 2003 .

[28]  James T. Kajiya,et al.  Anisotropic reflection models , 1985, SIGGRAPH.

[29]  Andrew Chi-Sing Leung,et al.  The plenoptic illumination function , 2002, IEEE Trans. Multim..

[30]  Shenchang Eric Chen,et al.  QuickTime VR: an image-based approach to virtual environment navigation , 1995, SIGGRAPH.

[31]  N. Ahmed,et al.  Discrete Cosine Transform , 1996 .

[32]  Donald P. Greenberg,et al.  Non-linear approximation of reflectance functions , 1997, SIGGRAPH.

[33]  Andrew Chi-Sing Leung,et al.  PCA-based compression for image-based relighting , 2003, 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698).

[34]  Chi-Wing Fu,et al.  Interactive Relighting of Panoramas , 2001, IEEE Computer Graphics and Applications.

[35]  Katsushi Ikeuchi,et al.  Eigen-texture method: Appearance compression based on 3D model , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).