Refinement of depth maps generated by low-cost depth sensors

With the introduction of the Microsoft Kinect into the gaming industry and release of Kinect-based application development kits, a whole new industry has evolved around the Kinect with applications for gaming, gesture recognition for controlling devices, 3-D communication, etc. coming to the fore. The popularity of the Kinect sensor can be attributed to its low cost and the real-time depth map generation capability. However, owing to Kinect's lack of sophistication, the depth maps generated have a lot of artifacts like poorly generated object boundaries and missing depth values and misalignment between the depth map and the color image. In this paper, a novel technique for depth map correction and post-processing is proposed. To fill the missing depth pixels in the generated depth map, we use a color-aware Gaussian-weighted averaging filter which estimates the missing depth values using the surrounding good depth pixels and by using the corresponding color image as a guide to prevent filtering across object boundaries. A depth map enhancement algorithm is then proposed based on multi-step upsampling-based anisotropic diffusion with structure aware weights using the SSIM distortion metric. This post-processing technique corrects and sharpens the boundaries of the objects of the depth map and ensures local depth smoothness within the objects. In all of the proposed techniques, an effort is made to reduce the complexity and keep the execution real-time.