Illumination animating and editing in a single picture using scene structure estimation

Abstract Editing the Illumination information in an image is a fundamental problem in image processing. In this paper, by estimating the scene structure, we propose a novel method to animate the illumination of a single input image. We first estimate the depth map of the input image by incorporating user interaction into the probabilistic inference model. Then we present a depth-aware intrinsic images decomposition method, which decomposes the image into specular shading, diffuse shading, and albedo. Combining the depth maps and shading maps, we develop a rendering-based optimization model to effectively estimate the positions and colors of multiple lights in a scene. With the estimated scene structure, including the depth map, reflectance, shading, as well as light positions and colors, we can animate and edit the illumination of the image via re-rendering the original image by changing the light configuration. Our method can effectively process images with complex shadow, multiple lights (including both point and area light sources) and specular spots. We show a variety of examples, including both indoor and outdoor environment to validate the effectiveness of the proposed method.