Joint upsampling and noise reduction for real-time depth map enhancement

An efficient system that upsamples depth map captured by Microsoft Kinect while jointly reducing the effect of noise is presented. The upsampling is carried by detecting and exploiting the piecewise locally planar structures of the downsampled depth map, based on corresponding high-resolution RGB image. The amount of noise is reduced by accumulating the downsampled data simultaneously. By benefiting from massively parallel computing capability of modern commodity GPUs, the system is able to maintain high frame rate. Our system is observed to produce the upsampled depth map that is very close to the original depth map both visually and mathematically.

[1]  Hideo Saito,et al.  Texture Overlay onto Non-rigid Surface using Commodity Depth Camera , 2012, VISAPP.

[2]  T. Fujii,et al.  Color based depth up-sampling for depth compression , 2010, 28th Picture Coding Symposium.

[3]  Sang Uk Lee,et al.  Superpixel-based depth image super-resolution , 2012, Electronic Imaging.

[4]  Dani Lischinski,et al.  Joint bilateral upsampling , 2007, SIGGRAPH 2007.

[5]  Marina L. Gavrilova,et al.  Transactions on Computational Science XII , 2011, Lecture Notes in Computer Science.

[6]  Sebastian Thrun,et al.  An Application of Markov Random Fields to Range Sensing , 2005, NIPS.

[7]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[8]  Pascal Fua,et al.  SLIC Superpixels Compared to State-of-the-Art Superpixel Methods , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Nassir Navab,et al.  Adaptive neighborhood selection for real-time surface normal estimation from organized point cloud data using integral images , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[10]  Zhigeng Pan,et al.  Real-Time Spatial and Depth Upsampling for Range Data , 2011, Trans. Comput. Sci..

[11]  Sebastian Thrun,et al.  A Noise‐aware Filter for Real‐time Depth Upsampling , 2008 .

[12]  Lifeng Sun,et al.  A novel upsampling scheme for depth map compression in 3DTV system , 2010, 28th Picture Coding Symposium.