Segmentation and Interpretation of Multicolored Objects with Highlights

Abstract This paper presents a segmentation system, based on a general framework for segmentation, that returns not only regions that correspond to coherent surfaces in an image, but also low-level interpretations of those regions' physical characteristics. This system is valid for images of piecewise uniform dielectric objects with highlights, moving it beyond the capabilities of previous physics-based segmentation algorithms which assume uniformly colored objects. This paper presents a summary of the complete system and focuses on two extensions of it that demonstrate its interpretive capacity and applicability to more complex scenes. The first extension provides interpretations of a scene by reasoning about the likelihood of different physical characteristics of simple image regions. The second extension allows the system to handle highlights within the general framework for segmentation. The resulting segmentations and interpretations more closely match our perceptions of objects since the resulting regions correspond to coherent surfaces, even when those surfaces have multiple colors and highlights.