Dense depth maps from sparse models and image coherence for augmented reality

A convincing combination of virtual and real data in an Augmented Reality (AR) application requires detailed 3D information about the real world scene. In many situations extensive model data is not available, while sparse representations such as outlines on a map exist. In this paper, we present a novel approach using such sparse 3D model data to seed automatic image segmentation and infer a dense depth map of an environment. Sparse 3D models of known landmarks, such as points and lines from GIS databases, are projected into a registered image and initialize 2D image segmentation at the projected locations in the image. For the segmentation we propose different techniques, which combine shape information, semantics given by the database, and the visual appearance in the referenced image. The resulting depth information of objects in the scene can be used in many applications, including occlusion handling, label placement, and 3D modeling.