Image-based rendering of diffuse, specular and glossy surfaces from a single image

In this paper, we present a new method to recover an approximation of the bidirectional reflectance distribution function (BRDF) of the surfaces present in a real scene. This is done from a single photograph and a 3D geometric model of the scene. The result is a full model of the reflectance properties of all surfaces, which can be rendered under novel illumination conditions with, for example, viewpoint modification and the addition of new synthetic objects. Our technique produces a reflectance model using a small number of parameters. These parameters nevertheless approximate the BRDF and allow the recovery of the photometric properties of diffuse, specular, isotropic or anisotropic textured objects. The input data are a geometric model of the scene including the light source positions and the camera properties, and a single image captured using this camera. Our algorithm generates a new synthetic image using classic rendering techniques, and a lambertian hypothesis about the reflectance model of the surfaces. Then, it iteratively compares the original image to the new one, and chooses a more complex reflectance model if the difference between the two images is greater than a user-defined threshold. We present several synthetic images that are compared to the original ones, and some possible applications in augmented reality.

[1]  Katsushi Ikeuchi,et al.  Illumination distribution from brightness in shadows: Adaptive estimation of illumination distribution with unknown reflectance properties in shadow regions , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[2]  J. Koenderink,et al.  Phenomenological description of bidirectional surface reflection , 1998 .

[3]  Loren C. Carpenter,et al.  The A -buffer, an antialiased hidden surface method , 1984, SIGGRAPH.

[4]  F. Durand.  Visibilité tridimensionnelle : étude analytique et apllications , 1999 .

[5]  James F. Blinn,et al.  Simulation of wrinkled surfaces , 1978, SIGGRAPH.

[6]  Steve Marschner,et al.  Image-Based BRDF Measurement Including Human Skin , 1999, Rendering Techniques.

[7]  E. Land,et al.  Lightness and retinex theory. , 1971, Journal of the Optical Society of America.

[8]  Larry Rudolph,et al.  A parallel scan conversion algorithm with anti-aliasing for a general-purpose ultracomputer , 1983, SIGGRAPH.

[9]  E. Adelson,et al.  The Plenoptic Function and the Elements of Early Vision , 1991 .

[10]  Donald P. Greenberg,et al.  The hemi-cube: a radiosity solution for complex environments , 1985, SIGGRAPH.

[11]  Katsushi Ikeuchi,et al.  Eigen-texture method: Appearance compression based on 3D model , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[12]  Steve Marschner,et al.  A practical model for subsurface light transport , 2001, SIGGRAPH.

[13]  Donald P. Greenberg,et al.  Improved Computational Methods for Ray Tracing , 1984, TOGS.

[14]  Takeo Kanade,et al.  Recovering shape in the presence of interreflections , 1991, Proceedings. 1991 IEEE International Conference on Robotics and Automation.

[15]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[16]  Ronen Basri,et al.  Lambertian reflectance and linear subspaces , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[17]  Steve Marschner,et al.  Inverse Lighting for Photography , 1997, CIC.

[18]  Michael Gervautz,et al.  An Image based Measurement System for Anisotropic Reflection , 1996, Comput. Graph. Forum.

[19]  Allen Gersho,et al.  Vector quantization and signal compression , 1991, The Kluwer international series in engineering and computer science.

[20]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[21]  Nelson L. Max,et al.  Bidirectional reflection functions from surface bump maps , 1987, SIGGRAPH.

[22]  Jitendra Malik,et al.  Modeling and editing real scenes with image-based techniques , 2000 .

[23]  Donald P. Greenberg,et al.  Modeling the interaction of light between diffuse surfaces , 1984, SIGGRAPH.

[24]  Julie Dorsey,et al.  Effic ient Re-rendering of Naturally Illuminated Environments , 1994 .

[25]  Gregory J. Ward,et al.  Measuring and modeling anisotropic reflection , 1992, SIGGRAPH.

[26]  Hans-Peter Seidel,et al.  Illuminating micro geometry based on precomputed visibility , 2000, SIGGRAPH.

[27]  Yoshiaki Shirai,et al.  Three-Dimensional Computer Vision , 1987, Symbolic Computation.

[28]  Hans-Peter Seidel,et al.  Automated texture registration and stitching for real world models , 2000, Proceedings the Eighth Pacific Conference on Computer Graphics and Applications.

[29]  Katsushi Ikeuchi,et al.  Illumination Distribution from Brightness in Shadows: Adaptive Estimation of Illumination Distribution with Unknown Reflectance Properties in Shadow Regions , 1999, ICCV.

[30]  K. Ikeuchi,et al.  Reflectance analysis under solar illumination , 1994, Proceedings of the Workshop on Physics-Based Modeling in Computer Vision.

[31]  Paul E. Debevec,et al.  Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography , 1998, SIGGRAPH '08.

[32]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[33]  Takeo Kanade,et al.  Determining shape and reflectance of hybrid surfaces by photometric sampling , 1989, IEEE Trans. Robotics Autom..

[34]  Michael J. Brooks,et al.  Shape and Source from Shading , 1985, IJCAI.

[35]  Norman J. McCormick,et al.  Inverse radiative transfer problems : a review , 1992 .

[36]  Jitendra Malik,et al.  Recognizing surfaces using three-dimensional textons , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[37]  Tien-Tsin Wong,et al.  Image-based Rendering with Controllable Illumination , 1997, Rendering Techniques.

[38]  Arthur Appel,et al.  Some techniques for shading machine renderings of solids , 1968, AFIPS Spring Joint Computing Conference.

[39]  Shree K. Nayar,et al.  Reflectance and texture of real-world surfaces , 1999, TOGS.

[40]  Michael J. Brooks,et al.  The variational approach to shape from shading , 1986, Comput. Vis. Graph. Image Process..

[41]  David S. Ebert,et al.  Data visualization , 2002, Comput. Graph..

[42]  Alexei A. Efros,et al.  Texture synthesis by non-parametric sampling , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[43]  Ned Greene,et al.  Environment Mapping and Other Applications of World Projections , 1986, IEEE Computer Graphics and Applications.

[44]  Harry Shum,et al.  Synthesizing bidirectional texture functions for real-world surfaces , 2001, SIGGRAPH.

[45]  Tom Davis,et al.  Opengl programming guide: the official guide to learning opengl , 1993 .

[46]  Anselmo Lastra,et al.  Increased photorealism for interactive architectural walkthroughs , 1999, SI3D.

[47]  A. Fournier,et al.  Common Illumination between Real and Computer Generated Scenes , 1992 .

[48]  Michael Garland,et al.  Surface simplification using quadric error metrics , 1997, SIGGRAPH.

[49]  Andrew S. Glassner,et al.  An introduction to ray tracing , 1989 .

[50]  Toyohisa Kaneko,et al.  Efficient and Handy Texture Mapping on 3D Surfaces , 1999, Comput. Graph. Forum.

[51]  S. P. Lloyd,et al.  Least squares quantization in PCM , 1982, IEEE Trans. Inf. Theory.

[52]  James F. Blinn,et al.  Models of light reflection for computer synthesized pictures , 1977, SIGGRAPH.

[53]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[54]  Baining Guo,et al.  Chaos Mosaic: Fast and Memory Efficient Texture Synthesis , 2000 .

[55]  Robert J. Woodham,et al.  Photometric method for determining surface orientation from multiple images , 1980 .

[56]  William H. Press,et al.  Numerical recipes in C , 2002 .

[57]  Edward H. Adelson,et al.  Estimating surface reflectance properties from images under unknown illumination , 2001, IS&T/SPIE Electronic Imaging.

[58]  Shoji Tominaga,et al.  Estimating Reflection Parameters from a Single Color Image , 2000, IEEE Computer Graphics and Applications.

[59]  Kosuke Sato,et al.  Determining Reflectance Properties of an Object Using Range and Brightness Images , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[60]  George Wolberg,et al.  Digital image warping , 1990 .

[61]  Gavin S. P. Miller,et al.  Lazy Decompression of Surface Light Fields for Precomputed Global Illumination , 1998, Rendering Techniques.

[62]  Pierre Poulin,et al.  Interactive Virtual Relighting and Remodeling of Real Scenes , 1999, Rendering Techniques.

[63]  Eero P. Simoncelli,et al.  Texture characterization via joint statistics of wavelet coefficient magnitudes , 1998, Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269).

[64]  Richard Szeliski,et al.  Layered depth images , 1998, SIGGRAPH.

[65]  William H. Press,et al.  The Art of Scientific Computing Second Edition , 1998 .

[66]  Paul Debevec,et al.  Modeling and Rendering Architecture from Photographs , 1996, SIGGRAPH 1996.

[67]  T. Caelli,et al.  Inverting an illumination model from range and intensity maps , 1994 .

[68]  Hans-Peter Seidel,et al.  Image-Based Reconstruction of Spatially Varying Materials , 2001 .

[69]  George Drettakis,et al.  Interactive Common Illumination for Computer Augmented Reality , 1997, Rendering Techniques.

[70]  K. Torrance,et al.  Image-Based BRDF Measurement , 1999 .

[71]  Zhengyou Zhang,et al.  Flexible camera calibration by viewing a plane from unknown orientations , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[72]  C. R. Hoffman,et al.  Illumination and Reflection Maps : Simulated Objects in Simulated and Real Environments Gene , 1984 .

[73]  F. E. Nicodemus,et al.  Geometrical considerations and nomenclature for reflectance , 1977 .

[74]  Andrea J. van Doorn,et al.  Bidirectional Reflection Distribution Function Expressed in Terms of Surface Scattering Modes , 1996, ECCV.

[75]  Katsushi Ikeuchi,et al.  Temporal-color space analysis of reflection , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[76]  R. Love.  Surface reflection model estimation from naturally illuminated image sequences , 1997 .

[77]  Donald P. Greenberg,et al.  A progressive refinement approach to fast radiosity image generation , 1988, SIGGRAPH.

[78]  Marc Rioux,et al.  Color Reflectance Modeling Using a Polychromatic Laser Range Sensor , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[79]  Adam Finkelstein,et al.  Lapped textures , 2000, SIGGRAPH.

[80]  Pat Hanrahan,et al.  Analysis of planar light fields from homogeneous convex curved surfaces under distant illumination , 2001, IS&T/SPIE Electronic Imaging.

[81]  Jitendra Malik,et al.  Recovering photometric properties of architectural scenes from photographs , 1998, SIGGRAPH.

[82]  Andrea J. van Doorn,et al.  Illuminance texture due to surface mesostructure , 1996 .

[83]  Paul Debevec,et al.  Inverse global illumination: Recovering re?ectance models of real scenes from photographs , 1998 .

[84]  K. Torrance,et al.  Polarization, Directional Distribution, and Off-Specular Peak Phenomena in Light Reflected from Roughened Surfaces , 1966 .

[85]  Gregory J. Ward,et al.  The RADIANCE lighting simulation and rendering system , 1994, SIGGRAPH.

[86]  Leonard McMillan,et al.  Plenoptic Modeling: An Image-Based Rendering System , 2023 .

[87]  Gabriel Taubin,et al.  Appying Shape from Lighting Variation to Bump Map Capture , 1997, Rendering Techniques.

[88]  Harry Shum,et al.  Rendering with concentric mosaics , 1999, SIGGRAPH.

[89]  Stephen H. Westin,et al.  A global illumination solution for general reflectance distributions , 1991, SIGGRAPH.

[90]  Jeremy S. De Bonet,et al.  Multiresolution sampling procedure for analysis and synthesis of texture images , 1997, SIGGRAPH.

[91]  P. Hanrahan,et al.  On the relationship between radiance and irradiance: determining the illumination from images of a convex Lambertian object. , 2001, Journal of the Optical Society of America. A, Optics, image science, and vision.

[92]  Katsushi Ikeuchi,et al.  Object shape and reflectance modeling from observation , 1997, SIGGRAPH.

[93]  Marc Levoy,et al.  Fast texture synthesis using tree-structured vector quantization , 2000, SIGGRAPH.

[94]  George Drettakis,et al.  Interactive Virtual Relighting of Real Scenes , 2000, IEEE Trans. Vis. Comput. Graph..

[95]  Katsushi Ikeuchi,et al.  Reflectance Analysis for 3D Computer Graphics Model Generation , 1996, CVGIP Graph. Model. Image Process..

[96]  Aaron F. Bobick,et al.  The direct computation of height from shading , 1991, Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[97]  B. Wilson,et al.  A diffusion theory model of spatially resolved, steady-state diffuse reflectance for the noninvasive determination of tissue optical properties in vivo. , 1992, Medical physics.

[98]  Pat Hanrahan,et al.  An efficient representation for irradiance environment maps , 2001, SIGGRAPH.

[99]  Hans-Peter Seidel,et al.  Towards interactive bump mapping with anisotropic shift-variant BRDFs , 2000, Workshop on Graphics Hardware.

[100]  J. Koenderink,et al.  Optical properties (bidirectional reflection distribution functions) of velvet. , 1998, Applied optics.

[101]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[102]  David Salesin,et al.  Surface light fields for 3D photography , 2000, SIGGRAPH.

[103]  Marc Levoy,et al.  The digital Michelangelo project: 3D scanning of large statues , 2000, SIGGRAPH.

[104]  F. A. Seiler,et al.  Numerical Recipes in C: The Art of Scientific Computing , 1989 .

[105]  L. M. M.-T..  Spherical Harmonics: an Elementary Treatise on Harmonic Functions, with Applications , 1928, Nature.

[106]  Konrad Klein,et al.  Texturing 3D Models of Real World Objects from Multiple Unregistered Photographic Views , 1999, Comput. Graph. Forum.

[107]  Hans-Peter Seidel,et al.  Image-Based BRDF Reconstruction , 1999 .

[108]  John A. Nelder,et al.  A Simplex Method for Function Minimization , 1965, Comput. J..

[109]  Donald P. Greenberg,et al.  Non-linear approximation of reflectance functions , 1997, SIGGRAPH.

[110]  J. MacQueen.  Some methods for classification and analysis of multivariate observations , 1967 .

[111]  Marc Levoy,et al.  A volumetric method for building complex models from range images , 1996, SIGGRAPH.

[112]  James J. Little,et al.  Reflectance function estimation and shape recovery from image sequence of a rotating object , 1995, Proceedings of IEEE International Conference on Computer Vision.

[113]  Edwin Earl Catmull,et al.  A subdivision algorithm for computer display of curved surfaces. , 1974 .

[114]  Pat Hanrahan,et al.  A signal-processing framework for inverse rendering , 2001, SIGGRAPH.

[115]  K. Torrance,et al.  Theory for off-specular reflection from roughened surfaces , 1967 .

[116]  T. Inui,et al.  Group theory and its applications in physics , 1990 .

[117]  Paul E. Debevec,et al.  Acquiring the reflectance field of a human face , 2000, SIGGRAPH.

[118]  Pat Hanrahan,et al.  A rapid hierarchical radiosity algorithm , 1991, SIGGRAPH.

[119]  Donald P. Greenberg,et al.  An Efficient Radiosity Approach for Realistic Image Synthesis , 1986, IEEE Computer Graphics and Applications.

[120]  Stephen H. Westin,et al.  Predicting reflectance functions from complex surfaces , 1992, SIGGRAPH.

[121]  Holly E. Rushmeier,et al.  Tone reproduction for realistic images , 1993, IEEE Computer Graphics and Applications.

[122]  James R. Bergen,et al.  Pyramid-based texture analysis/synthesis , 1995, Proceedings., International Conference on Image Processing.

[123]  Shree K. Nayar,et al.  Histogram model for 3D textures , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[124]  大野 茂,et al.  Polarization , 2006, A First Course in Laboratory Optics.