Modeling and editing real scenes with image-based techniques

Image-based modeling and rendering techniques greatly advanced the level of photorealism in computer graphics. They were originally proposed to accelerate rendering with the ability to vary viewpoint only. This thesis focuses on capturing and modeling real scenes for novel visual interactions such as varying lighting condition and scene configuration in addition to viewpoint. This work can lead to applications such as virtual navigation of a real scene, interaction with the scene, novel scene composition, interior lighting design, and augmented reality. This thesis has two important parts. The first part includes the techniques to extract an object-level representation of a real scene which can be rendered with modifications to the existing spatial configuration. The key components here involve automatic algorithms to segment the geometry from range images into distinct surfaces, and register texture from radiance images with the geometry. The top-down segmentation algorithm uses a pairwise similarity measurement to recursively partition a point set into a binary tree with individual surfaces as leaves. Our image registration technique can automatically find the camera poses for arbitrary position and orientation relative to the geometry. The second part includes the inverse global illumination technique which refers to recovering reflectance models of various materials present in a real mutual illumination environment. The method's input is a geometric model of the scene and a set of calibrated photographs taken with known light source positions. The result is a lighting-independent model of the scene's geometry and reflectance properties, which can be rendered under novel lighting conditions using traditional graphics methods. The underlying philosophy is using low-parameter BRDF models and solving optimization problems to recover the parameters. Synthetic images rendered using recovered BRDF models are comparable to real photographs. Complementary to the above two parts, this thesis also presents a technique for recovering illumination conditions of outdoor scenes, algorithms for texture map synthesis from real photographs and texture map compression for efficient conventional texture-mapping, and a visibility algorithm for projective texture-mapping.