A new multi-lateral filter for real-time depth enhancement

We present an adaptive multi-lateral filter for real-time low-resolution depth map enhancement. Despite the great advantages of Time-of-Flight cameras in 3-D sensing, there are two main drawbacks that restricts their use in a wide range of applications; namely, their fairly low spatial resolution, compared to other 3-D sensing systems, and the high noise level within the depth measurements. We therefore propose a new data fusion method based upon a bilateral filter. The proposed filter is an extension the pixel weighted average strategy for depth sensor data fusion. It includes a new factor that allows to adaptively consider 2-D data or 3-D data as guidance information. Consequently, unwanted artefacts such as texture copying get almost entirely eliminated, outperforming alternative depth enhancement filters. In addition, our algorithm can be effectively and efficiently implemented for real-time applications.

[1]  Narendra Ahuja,et al.  Real-time O(1) bilateral filtering , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  A. Frick,et al.  3D-TV LDV content generation with a hybrid ToF-multicamera RIG , 2010, 2010 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[3]  Björn E. Ottersten,et al.  Pixel weighted average strategy for depth sensor data fusion , 2010, 2010 IEEE International Conference on Image Processing.

[4]  Ruigang Yang,et al.  Fusion of time-of-flight depth and stereo for high accuracy depth maps , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Franz Franchetti,et al.  Fast bilateral filtering by adapting block size , 2010, 2010 IEEE International Conference on Image Processing.

[6]  Dani Lischinski,et al.  Joint bilateral upsampling , 2007, SIGGRAPH 2007.

[7]  Mongi A. Abidi,et al.  Spatial and Temporal Enhancement of Depth Images Captured by a Time-of-Flight Depth Sensor , 2010, 2010 20th International Conference on Pattern Recognition.

[8]  Carme Torras,et al.  Exploitation of time-of-flight (ToF) cameras , 2010 .

[9]  Sebastian Thrun,et al.  An Application of Markov Random Fields to Range Sensing , 2005, NIPS.

[10]  Roberto Manduchi,et al.  Bilateral filtering for gray and color images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[11]  R. Lange,et al.  Solid-state time-of-flight range camera , 2001 .

[12]  Frédo Durand,et al.  A Fast Approximation of the Bilateral Filter Using a Signal Processing Approach , 2006, ECCV.

[13]  Reinhard Koch,et al.  Time-of-Flight Sensors in Computer Graphics , 2009, Eurographics.

[14]  K. Natroshvili,et al.  Real time pedestrian detection by fusing PMD and CMOS cameras , 2008, 2008 IEEE Intelligent Vehicles Symposium.

[15]  Ryan Crabb,et al.  Real-time foreground segmentation via range and color imaging , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[16]  Fatih Porikli,et al.  Constant time O(1) bilateral filtering , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Sebastian Thrun,et al.  A Noise‐aware Filter for Real‐time Depth Upsampling , 2008 .

[18]  Andrew Y. Ng,et al.  Integrating Visual and Range Data for Robotic Object Detection , 2008, ECCV 2008.

[19]  Ruigang Yang,et al.  Spatial-Depth Super Resolution for Range Images , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.