Real-time processing of depth and color video streams to improve the reliability of depth maps

Depth is a useful information in vision to understand the geometrical properties of an environment. Depth is traditionally computed in terms of a disparity map acquired by a stereoscopic system but, over the last few years, several manufacturers have released single-lens cameras that directly capture depth information (also called range). This is an important technological breakthrough although range signals remain difficult to handle in practice, due to many reasons (low resolution, noise, low framerate, . . . ). Practitioners still struggle to use range data in their applications. The purpose of this paper is to give a brief introduction to range data (captured with a camera), discuss common limitations, and propose techniques to cope with difficulties typically encountered with range cameras. These techniques are based on a simultaneous view of the scene by a color and a depth camera that are combined to improve their interpretation in real time.