A Theory Of Frequency Domain Invariants: Spherical Harmonic Identities for BRDF/Lighting Transfer and Image Consistency

This paper develops a theory of frequency domain invariants in computer vision. We derive novel identities using spherical harmonics, which are the angular frequency domain analog to common spatial domain invariants such as reflectance ratios. These invariants are derived from the spherical harmonic convolution framework for reflection from a curved surface. Our identities apply in a number of canonical cases, including single and multiple images of objects under the same and different lighting conditions. One important case we consider is two different glossy objects in two different lighting environments. For this case, we derive a novel identity, independent of the specific lighting configurations or BRDFs, that allows us to directly estimate the fourth image if the other three are available. The identity can also be used as an invariant to detect tampering in the images. Although this paper is primarily theoretical, it has the potential to lay the mathematical foundations for two important practical applications. First, we can develop more general algorithms for inverse rendering problems, which can directly relight and change material properties by transferring the BRDF or lighting from another object or illumination. Second, we can check the consistency of an image to detect tampering or image splicing.

[1]  Steve Marschner,et al.  Inverse Lighting for Photography , 1997, CIC.

[2]  Frédo Durand,et al.  A gentle introduction to bilateral filtering and its applications , 2007, SIGGRAPH Courses.

[3]  Ira Kemelmacher-Shlizerman,et al.  Photometric Stereo with General, Unknown Lighting , 2006, International Journal of Computer Vision.

[4]  David Salesin,et al.  Image Analogies , 2001, SIGGRAPH.

[5]  Steven M. Seitz,et al.  Single-view modelling of free-form scenes , 2002, Comput. Animat. Virtual Worlds.

[6]  K. Torrance,et al.  Image-Based BRDF Measurement , 1999 .

[7]  Matthew Turk,et al.  A Morphable Model For The Synthesis Of 3D Faces , 1999, SIGGRAPH.

[8]  J. Todd.  Book Review: Digital image processing (second edition). By R. C. Gonzalez and P. Wintz, Addison-Wesley, 1987. 503 pp. Price: £29.95. (ISBN 0-201-11026-1) , 1988 .

[9]  Ruigang Yang,et al.  BRDF Invariant Stereo Using Light Transport Constancy , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Hany Farid,et al.  Exposing digital forgeries by detecting inconsistencies in lighting , 2005, MM&Sec '05.

[11]  Zicheng Liu,et al.  Face relighting with radiance environment maps , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[12]  David J. Kriegman,et al.  Specularity Removal in Images and Videos: A PDE Approach , 2006, ECCV.

[13]  Pat Hanrahan,et al.  A signal-processing framework for inverse rendering , 2001, SIGGRAPH.

[14]  Lei Zhang,et al.  Face synthesis and recognition from a single image under arbitrary unknown lighting using a spherical harmonic basis morphable model , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[15]  Li Zhang,et al.  Single view modeling of free-form scenes , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[16]  Rongrong Wang,et al.  Detecting doctored images using camera response normality and consistency , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[17]  Pat Hanrahan,et al.  Frequency space environment map rendering , 2002, SIGGRAPH.

[18]  Katsushi Ikeuchi,et al.  Illumination Distribution from Brightness in Shadows: Adaptive Estimation of Illumination Distribution with Unknown Reflectance Properties in Shadow Regions , 1999, ICCV.

[19]  Szymon Rusinkiewicz,et al.  A New Change of Variables for Efficient BRDF Representation , 1998, Rendering Techniques.

[20]  Shree K. Nayar,et al.  A class of photometric invariants: separating material from shape and illumination , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[21]  Steven M. Seitz,et al.  Example-based photometric stereo: shape reconstruction with general, varying BRDFs , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Ronen Basri,et al.  Lambertian reflectance and linear subspaces , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[23]  Jan Kautz,et al.  Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments , 2002 .

[24]  Stephen H. Westin,et al.  Image-based bidirectional reflectance distribution function measurement. , 2000, Applied optics.

[25]  Ravi Ramamoorthi,et al.  A Theory of Spherical Harmonic Identities for BRDF/Lighting Transfer and Image Consistency , 2006, ECCV.

[26]  Ronen Basri,et al.  Dense shape reconstruction of a moving object under arbitrary, unknown lighting , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[27]  Shih-Fu Chang,et al.  Blind detection of photomontage using higher order statistics , 2004, 2004 IEEE International Symposium on Circuits and Systems (IEEE Cat. No.04CH37512).

[28]  Stefano Soatto,et al.  Multi-view stereo beyond Lambert , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[29]  Norbert Wiener,et al.  Extrapolation, Interpolation, and Smoothing of Stationary Time Series , 1964 .

[30]  Shree K. Nayar,et al.  Reflectance based object recognition , 1996, International Journal of Computer Vision.