Unsupervised visual hull extraction in space, time and light domains

Abstract This paper presents an unsupervised image segmentation approach for obtaining a set of silhouettes along with the visual hull (VH) of an object observed from multiple viewpoints. The proposed approach can deal with mostly any type of appearance characteristics such as texture, similar background color, shininess, transparency besides other phenomena such as shadows and color bleeding. Compared to more classical methods for silhouette extraction from multiple views, for which certain assumptions are made on the object or scene, neither the background nor the object appearance properties are modeled. The only assumption is the constancy of the unknown background for a given camera viewpoint while the object is under motion. The principal idea of the method is the estimation of the temporal evolution of each pixel over time which provides a stability measurement and leads to its associated background likelihood. In order to cope with shadows and self-shadows, an object is captured under different lighting conditions. Furthermore, the information from the space, time and lighting domains is exploited and merged based on a MRF framework and the constructed energy function is minimized via graph cut. Experiments are performed on a light stage where the object is set on a turntable and is observed from calibrated viewpoints on a hemisphere around the object. Real data experiments show that the proposed approach allows for robust and efficient VH reconstruction of a variety of challenging objects.