Spectral analysis for sampling image-based rendering data

Image-based rendering (IBR) has become a very active research area in recent years. The spectral analysis problem for IBR has not been completely solved. In this paper, we present a new method to parameterize the problem, which is applicable for general-purpose IBR spectral analysis. We notice that any plenoptic function is generated by light ray emitted/reflected/refracted from the object surface. We introduce the surface plenoptic function (SPF), which represents the light rays starting from the object surface. Given that radiance along a light ray does not change unless the light ray is blocked, SPF reduces the dimension of the original plenoptic function to 6D. We are then able to map or transform the SPF to IBR representations captured along any camera trajectory. Assuming some properties on the SPF, we can analyze the properties of IBR for generic scenes such as scenes with Lambertian or non-Lambertian surfaces and scenes with or without occlusions, and for different sampling strategies such as lightfield/concentric mosaic. We find that in most cases, even though the SPF may be band-limited, the frequency spectrum of IBR is not band-limited. We show that non-Lambertian reflections, depth variations and occlusions can all broaden the spectrum, with the latter two being more significant. SPF is defined for scenes with known geometry. When the geometry is unknown, spectral analysis is still possible. We show that with the "truncating windows" analysis and some conclusions obtained with SPF, the spectrum expansion caused by non-Lambertian reflections and occlusions can be quantatively estimated, even when the scene geometry is not explicitly known. Given the spectrum of IBR, we also study how to sample IBR data more efficiently. Our analysis is based on the generalized periodic sampling theory with arbitrary geometry. We show that the sampling efficiency can be up to twice of that when we use rectangular sampling. The advantages and disadvantages of generalized periodic sampling for IBR are also discussed.

[1]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[2]  Harry Shum,et al.  Plenoptic sampling , 2000, SIGGRAPH.

[3]  P. Vaidyanathan.  Multirate Systems And Filter Banks , 1992 .

[4]  Truong Q. Nguyen,et al.  Eigenfilters: A new approach to least-squares FIR filter design and applications including Nyquist filters , 1987 .

[5]  Gavin S. P. Miller,et al.  Lazy Decompression of Surface Light Fields for Precomputed Global Illumination , 1998, Rendering Techniques.

[6]  Dan E. Dudgeon,et al.  Multidimensional Digital Signal Processing , 1983 .

[7]  P. Debevec,et al.  Image-based modeling, rendering, and lighting , 2002, IEEE Computer Graphics and Applications.

[8]  E. Adelson,et al.  The Plenoptic Function and the Elements of Early Vision , 1991 .

[9]  Tsuhan Chen,et al.  Unified eigenfilter approach: with applications to spectral/spatial filtering , 1993, 1993 IEEE International Symposium on Circuits and Systems.

[10]  Harry Shum,et al.  Relighting with the Reflected Irradiance Field: Representation, Sampling and Reconstruction , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[11]  Leonard McMillan,et al.  Plenoptic Modeling: An Image-Based Rendering System , 2023 .

[12]  Harry Shum,et al.  Rendering with concentric mosaics , 1999, SIGGRAPH.

[13]  Harry Shum,et al.  A spectral analysis for light field rendering , 2000, Proceedings 2000 International Conference on Image Processing (Cat. No.00CH37101).

[14]  Harry Shum,et al.  On the number of samples needed in light field rendering with constant-depth assumption , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[15]  Andrew Chi-Sing Leung,et al.  The plenoptic illumination function , 2002, IEEE Trans. Multim..

[16]  Pat Hanrahan,et al.  A signal-processing framework for inverse rendering , 2001, SIGGRAPH.

[17]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[18]  David Salesin,et al.  Surface light fields for 3D photography , 2000, SIGGRAPH.

[19]  Jean Ponce,et al.  Computer Vision: A Modern Approach , 2002 .