Sparse lumigraph relighting by illumination and reflectance estimation from multi-view images

We present a novel relighting approach that does not assume that the illumination is known or controllable. Instead, we estimate the illumination and texture from given multi-view images captured under a single illumination setting, given the object shape. We rely on the viewpoint-dependence of surface reflectance to resolve the usual texture-illumination ambiguity. The task of obtaining the illumination and texture models is formulated as the decomposition of the observed surface radiance tensor into the product of a light transport tensor, and illumination and texture matrices. We estimate both the illumination and texture at the same time by solving a system of bilinear equations. To reduce estimation error due to imperfect input surface geometry, we also perform a multi-scale discrete search on the specular surface normal. Our results on synthetic and real data indicate that we can estimate the illumination, the diffuse as well as the specular components of the surface texture map (up to a global scaling ambiguity). Our approach allows more flexibilities in rendering novel images, such as view changing, and light and texture editing.