Tales of shape and radiance in multiview stereo

To what extent can three-dimensional shape and radiance be inferred from a collection of images? Can the two be estimated separately while retaining optimality? How should the optimality criterion be computed? When is it necessary to employ an explicit model of the reflectance properties of a scene? In this paper we introduce a separation principle for shape and radiance estimation that applies to Lambertian scenes and holds for any choice of norm. When the scene is not Lambertian, however, shape cannot be decoupled from radiance, and therefore matching image-to-image is not possible directly. We employ a rank constraint on the radiance tensor, which is commonly used in computer graphics, and construct a novel cost functional whose minimization leads to an estimate of both shape and radiance for nonLambertian objects, which we validate experimentally.

[1]  Berthold K. P. Horn.  Robot vision , 1986, MIT electrical engineering and computer science series.

[2]  J. Sethian,et al.  Fronts propagating with curvature-dependent speed: algorithms based on Hamilton-Jacobi formulations , 1988 .

[3]  S. Osher,et al.  Algorithms Based on Hamilton-Jacobi Formulations , 1988 .

[4]  J J Koenderink,et al.  Affine structure from motion. , 1991, Journal of the Optical Society of America. A, Optics and image science.

[5]  O. Faugeras.  Stratification of three-dimensional vision: projective, affine, and metric representations , 1995 .

[6]  Olivier D. Faugeras,et al.  Variational principles, surface evolution, PDEs, level set methods, and the stereo problem , 1998, IEEE Trans. Image Process..

[7]  Paul Debevec,et al.  Inverse global illumination: Recovering re?ectance models of real scenes from photographs , 1998 .

[8]  A. Rau.  Variational Principles , 2021, Classical Mechanics.

[9]  David J. Kriegman,et al.  Beyond Lambert: reconstructing surfaces with arbitrary BRDFs , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[10]  Stefano Soatto,et al.  Variational multiframe stereo in the presence of specular reflections , 2002, Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission.

[11]  Wei-Chao Chen,et al.  Light field mapping: efficient representation and hardware rendering of surface light fields , 2002, SIGGRAPH.

[12]  Stefano Soatto,et al.  Variational methods for shape reconstruction in computer vision , 2003 .

[13]  Stefano Soatto,et al.  Multi-view stereo beyond Lambert , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[14]  Takeo Kanade,et al.  When Is the Shape of a Scene Unique Given Its Light-Field: A Fundamental Theorem of 3D Vision? , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  Todd E. Zickler,et al.  Helmholtz Stereopsis: Exploiting Reciprocity for Surface Reconstruction , 2002, International Journal of Computer Vision.

[16]  David J. Kriegman,et al.  The Bas-Relief Ambiguity , 2004, International Journal of Computer Vision.

[17]  Daniel Snow,et al.  Determining Generative Models of Objects Under Varying Illumination: Shape and Albedo from Multiple Images Using SVD and Integrability , 1999, International Journal of Computer Vision.