Depth images super-resolution: An iterative approach

Among all the techniques for 3D acquisition, stereo vision systems are the most common. More recently, Time-of-Flight (ToF) range cameras have been introduced. They allow real-time depth estimation in conditions where stereo does not work well. Unfortunately, ToF sensors still have a limited resolution (e.g., 200 × 200 pixels). The goal of this paper is to combine the information from the ToF with one or two standard cameras, in order to obtain a highresolution depth image. To this end, we propose a new bilateral filter for depth maps up-sampling, which exploits the additional information provided by a high-resolution color image. Moreover, we present an entire framework for the super-resolution of a real ToF depth map with a single camera color image. The algorithm allows to enhance the resolution of the ToF camera depth image up to 1920 × 1080 pixels.

[1]  Sebastian Schwarz,et al.  A Weighted Optimization Approach to Time-of-Flight Sensor Fusion , 2014, IEEE Transactions on Image Processing.

[2]  S. E. Ghobadi,et al.  First steps in enhancing 3D vision technique using 2D/3D sensors , 2006 .

[3]  Dani Lischinski,et al.  Joint bilateral upsampling , 2007, ACM Trans. Graph..

[4]  Gang Yang,et al.  Depth super resolution using bilateral filter , 2011, 2011 4th International Congress on Image and Signal Processing.

[5]  Fulvio Rinaudo,et al.  SR-4000 and CamCube3.0 Time of Flight (ToF) Cameras: Tests and Comparison , 2012, Remote. Sens..

[6]  F. Pukelsheim.  The Three Sigma Rule , 1994 .

[7]  Pietro Zanuttigh,et al.  A Novel Interpolation Scheme for Range Data with Side Information , 2009, 2009 Conference for Visual Media Production.

[8]  Michael F. Cohen,et al.  Digital photography with flash and no-flash image pairs , 2004, ACM Trans. Graph..

[9]  Robert Mahony,et al.  Statistical analysis of measurement processes for time-of-flight cameras , 2009, Optical Engineering + Applications.

[10]  Stefano Mattoccia,et al.  Locally Consistent ToF and Stereo Data Fusion , 2012, ECCV Workshops.

[11]  Sebastian Thrun,et al.  An Application of Markov Random Fields to Range Sensing , 2005, NIPS.

[12]  Roberto Manduchi,et al.  Bilateral filtering for gray and color images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[13]  A. Vianello.  Depth Super-Resolution with Hybrid Camera System , 2013 .

[14]  S. Foix,et al.  Lock-in Time-of-Flight (ToF) Cameras: A Survey , 2011, IEEE Sensors Journal.

[15]  Carlo Dal Mutto.  Acquisition and Processing of ToF and Stereo data , 2013 .

[16]  Thierry Oggier,et al.  CCD / CMOS Lock-In Pixel for Range Imaging : Challenges , Limitations and State-ofthe-Art , 2005 .

[17]  F. Durand,et al.  Flash photography enhancement via intrinsic relighting , 2004, ACM Trans. Graph..

[18]  Ruigang Yang,et al.  Spatial-Depth Super Resolution for Range Images , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Bernd Jähne,et al.  A Physical Model of Time-of-Flight 3D Imaging Systems, Including Suppression of Ambient Light , 2009, Dyn3D.

[20]  Alice Biber,et al.  Time-of-flight range imaging with a custom solid state image sensor , 1999, Industrial Lasers and Inspection.

[21]  Heiko Hirschmüller,et al.  Evaluation of Cost Functions for Stereo Matching , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.