Guided optimization framework for the fusion of time-of-flight with stereo depth

Abstract. The fusion of depth acquired actively with the depth estimated passively proved its significance as an improvement strategy for gaining depth. This combination allows us to benefit from two sources of modalities such that they complement each other. To fuse two sensor data into a more accurate depth map, we must consider the limitations of active sensing such as low lateral resolution while combining it with a passive depth map. We present an approach for the fusion of active time-of-flight depth and passive stereo depth in an accurate way. We propose a multimodal sensor fusion strategy that is based on a weighted energy optimization problem. The weights are generated as a result of combining the edge information from a texture map and active and passive depth maps. The objective evaluation of our fusion algorithm shows an improved accuracy of the generated depth map in comparison with the depth map of every single modality and with the results of other fusion methods. Additionally, a visual comparison of our result shows a better recovery on the edges considering the wrong depth values estimated in passive stereo. Moreover, the left and right consistency check on the result illustrates the ability of our approach to consistently fuse sensors.

[1]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[2]  Sebastian Thrun,et al.  A Noise‐aware Filter for Real‐time Depth Upsampling , 2008 .

[3]  Stefano Mattoccia,et al.  Reliable Fusion of ToF and Stereo Depth Driven by Confidence Measures , 2016, ECCV.

[4]  Sebastian Schwarz,et al.  A Weighted Optimization Approach to Time-of-Flight Sensor Fusion , 2014, IEEE Transactions on Image Processing.

[5]  Xing Mei,et al.  On building an accurate stereo matching system on graphics hardware , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[6]  Robert P. W. Duin,et al.  Fast percentile filtering , 1986, Pattern Recognit. Lett..

[7]  Sebastian Thrun,et al.  Upsampling range data in dynamic environments , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[8]  Ruigang Yang,et al.  Spatial-Depth Super Resolution for Range Images , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[9]  Marcus A. Magnor,et al.  A Survey on Time-of-Flight Stereo Fusion , 2013, Time-of-Flight and Depth Imaging.

[10]  Michael S. Brown,et al.  High-Quality Depth Map Upsampling and Completion for RGB-D Cameras , 2014, IEEE Transactions on Image Processing.

[11]  Sebastian Thrun,et al.  An Application of Markov Random Fields to Range Sensing , 2005, NIPS.

[12]  Thierry Oggier,et al.  CCD / CMOS Lock-In Pixel for Range Imaging : Challenges , Limitations and State-ofthe-Art , 2005 .

[13]  Yutaka Ishibashi,et al.  Weighted Joint Bilateral Filter with Slope Depth Compensation Filter for Depth Map Refinement , 2013, VISAPP.

[14]  M. Meister.  Problems And Solutions In Introductory And Advanced Matrix Calculus , 2016 .

[15]  Sebastian Schwarz,et al.  Temporal consistent depth map upscaling for 3DTV , 2014, Electronic Imaging.

[16]  Juho Kannala,et al.  Joint Depth and Color Camera Calibration with Distortion Correction , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Guido M. Cortelazzo,et al.  Probabilistic ToF and Stereo Data Fusion Based on Mixed Pixels Measurement Models , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Bernd Jähne,et al.  Theoretical and experimental error analysis of continuous-wave time-of-flight range cameras , 2009, Optical Engineering.

[19]  Stefano Mattoccia,et al.  Locally Consistent ToF and Stereo Data Fusion , 2012, ECCV Workshops.

[20]  Ludovico Minto,et al.  Time-of-Flight and Structured Light Depth Cameras , 2016, Springer International Publishing.

[21]  Rasmus Larsen,et al.  Fusion of stereo vision and Time-Of-Flight imaging for improved 3D estimation , 2008, Int. J. Intell. Syst. Technol. Appl..

[22]  Dani Lischinski,et al.  Joint bilateral upsampling , 2007, SIGGRAPH 2007.

[23]  Pietro Zanuttigh,et al.  Joint segmentation of color and depth data based on splitting and merging driven by surface fitting , 2018, Image Vis. Comput..

[24]  R. Lange,et al.  Solid-state time-of-flight range camera , 2001 .

[25]  Radu Horaud,et al.  Fusion of Range and Stereo Data for High-Resolution Scene-Modeling , 2015, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Heiko Hirschmüller,et al.  Stereo Processing by Semiglobal Matching and Mutual Information , 2008, IEEE Trans. Pattern Anal. Mach. Intell..

[27]  Paolo Favaro,et al.  Recovering thin structures via nonlocal-means regularization with application to depth from defocus , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[28]  Shmuel Peleg,et al.  Multi-sensor super-resolution , 2002, Sixth IEEE Workshop on Applications of Computer Vision, 2002. (WACV 2002). Proceedings..

[29]  Timo Schairer,et al.  Fusion of range and color images for denoising and resolution enhancement with a non-local filter , 2010, Comput. Vis. Image Underst..

[30]  Sebastian Schwarz,et al.  Time-of-flight sensor fusion with depth measurement reliability weighting , 2014, 2014 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[31]  Joachim Keinert,et al.  A Novel Confidence Measure for Disparity Maps by Pixel-Wise Cost Function Analysis , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[32]  Gianluca Agresti,et al.  Deep Learning for Confidence Information in Stereo and ToF Data Fusion , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[33]  Antonio Torralba,et al.  Properties and applications of shape recipes , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..