Fusion of Time-of-Flight and Stereo for Disambiguation of Depth Measurements

The complementary nature of time-of-flight and stereo has led to their fusion systems, providing high quality depth maps robustly against depth bias and random noise of the time-of-flight camera as well as the lack of scene texture. This paper shows that the fusion system is also effective for disambiguating time-of-flight depth measurements caused by phase wrapping, which records depth values that are much less than their actual values if the scene points are farther than a certain maximum range. To recover the unwrapped depth map, we build a Markov random field based on a constraint that an accurately unwrapped depth value should minimize the dissimilarity between its projections on the stereo images. The unwrapped depth map is then adapted to stereo matching, reducing the matching ambiguity and enhancing the depth quality in textureless regions. Through experiments we show that the proposed method extends the range use of the time-of-flight camera, delivering unambiguous depth maps of real scenes.

[1]  Hubert Roth,et al.  Combining PMD and Stereo camera for Motion Estimation of a Mobile Robot , 2008 .

[2]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[3]  Roberto Manduchi,et al.  Bilateral filtering for gray and color images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[4]  Reinhard Koch,et al.  Dynamic 3D Imaging, DAGM 2009 Workshop, Dyn3D 2009, Jena, Germany, September 9, 2009. Proceedings , 2009, Dyn3D.

[5]  Klaus-Dieter Kuhnert,et al.  Fusion of Stereo-Camera and PMD-Camera Data for Real-Time Suited Precise 3D Environment Reconstruction , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[6]  Reinhard Koch,et al.  Time-of-Flight sensor calibration for accurate range sensing , 2010, Comput. Vis. Image Underst..

[7]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[8]  Ruigang Yang,et al.  Reliability Fusion of Time-of-Flight Depth and Stereo Geometry for High Quality Depth Maps , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Sang Uk Lee,et al.  Illumination and camera invariant stereo matching , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  B. Buttgen,et al.  Demodulation Pixel Based on Static Drift Fields , 2006, IEEE Transactions on Electron Devices.

[11]  Chang-Yeong Kim,et al.  Range unfolding for Time-of-Flight depth cameras , 2010, 2010 IEEE International Conference on Image Processing.

[12]  In-So Kweon,et al.  Adaptive Support-Weight Approach for Correspondence Search , 2006, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  Dale A. Carnegie,et al.  Maximizing precision over extended unambiguous range for TOF range imaging systems , 2010, 2010 IEEE Instrumentation & Measurement Technology Conference Proceedings.

[14]  Marc Alexa,et al.  Combining Time-Of-Flight depth and stereo images without accurate extrinsic calibration , 2008, Int. J. Intell. Syst. Technol. Appl..

[15]  Ruigang Yang,et al.  Spatial-Temporal Fusion for High Accuracy Depth Maps Using Dynamic MRFs , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Marc Alexa,et al.  Depth Imaging by Combining Time-of-Flight and On-Demand Stereo , 2009, Dyn3D.

[17]  William H. Press,et al.  Numerical recipes in C , 2002 .

[18]  Sven Behnke,et al.  Probabilistic Phase Unwrapping for Time-of-Flight Cameras , 2010, ISR/ROBOTIK.

[19]  Michael J. Cree,et al.  Resolving depth-measurement ambiguity with commercially available range imaging cameras , 2010, Electronic Imaging.

[20]  Reinhard Koch,et al.  A Combined Approach for Estimating Patchlets from PMD Depth Images and Stereo Intensity Images , 2007, DAGM-Symposium.

[21]  Chang-Yeong Kim,et al.  Discrete and continuous optimizations for depth image super-resolution , 2012, Electronic Imaging.

[22]  Rasmus Larsen,et al.  Fusion of stereo vision and Time-Of-Flight imaging for improved 3D estimation , 2008, Int. J. Intell. Syst. Technol. Appl..

[23]  Olga Veksler,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[24]  Andreas Kolb,et al.  Sub-pixel data fusion and edge-enhanced distance refinement for 2D/3D images , 2008, Int. J. Intell. Syst. Technol. Appl..

[25]  Radu Horaud,et al.  Projective alignment of range and parallax data , 2011, CVPR 2011.

[26]  Byongmin Kang,et al.  Parametric model-based noise reduction for ToF depth sensors , 2012, Electronic Imaging.

[27]  S. Birchfiled.  A Pixel Dissimilarity Measure That Is Insensitive to Image Sampling , 1998 .

[28]  Sven Behnke,et al.  Multi-frequency Phase Unwrapping for Time-of-Flight cameras , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[29]  Young Min Kim,et al.  Multi-view image and ToF sensor fusion for dense 3D reconstruction , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[30]  S. Burak Gokturk,et al.  A Time-Of-Flight Depth Sensor - System Description, Issues and Solutions , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.