Confidence Estimation for ToF and Stereo Sensors and Its Application to Depth Data Fusion

Time-of-Flight (ToF) sensors and stereo vision systems are two widely used technologies for depth estimation. Due to their rather complementary strengths and limitations, the two sensors are often combined to infer more accurate depth maps. A key research issue in this field is how to estimate the reliability of the sensed depth data. While this problem has been widely studied for stereo systems, it has been seldom considered for ToF sensors. Therefore, starting from the work done for stereo data, in this paper, we firstly introduce novel confidence estimation techniques for ToF data. Moreover, we also show how by using learning-based confidence metrics jointly trained on the two sensors yields better performance. Finally, deploying different fusion frameworks, we show how confidence estimation can be exploited in order to guide the fusion of depth data from the two sensors. Experimental results show how accurate confidence cues allow outperforming state-of-the-art data fusion schemes even with the simplest fusion strategies known in the literature.

[1]  Stefano Mattoccia,et al.  Reliable Fusion of ToF and Stereo Depth Driven by Confidence Measures , 2016, ECCV.

[2]  Rasmus Larsen,et al.  Fusion of stereo vision and Time-Of-Flight imaging for improved 3D estimation , 2008, Int. J. Intell. Syst. Technol. Appl..

[3]  Marc Pollefeys,et al.  Patch Based Confidence Prediction for Dense Disparity Map , 2016, BMVC.

[4]  Gianluca Agresti,et al.  Stereo and ToF data fusion by learning from synthetic data , 2019, Inf. Fusion.

[5]  Gianluca Agresti,et al.  Deep Learning for Confidence Information in Stereo and ToF Data Fusion , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[6]  Cheolkon Jung,et al.  Variational fusion of time-of-flight and stereo data using edge selective joint filtering , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[7]  Stefano Mattoccia,et al.  Learning to Predict Stereo Reliability Enforcing Local Consistency of Confidence Maps , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Tim Weyrich,et al.  Capturing Time-of-Flight data with confidence , 2011, CVPR 2011.

[9]  Stefano Mattoccia,et al.  Quantitative Evaluation of Confidence Measures in a Machine Learning World , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[10]  Cheolkon Jung,et al.  Variational Fusion of Time-of-Flight and Stereo Data for Depth Estimation Using Edge-Selective Joint Filtering , 2018, IEEE Transactions on Multimedia.

[11]  Stefano Mattoccia,et al.  Beyond Local Reasoning for Stereo Confidence Estimation with Deep Learning , 2018, ECCV.

[12]  Min H. Kim,et al.  DeepToF: off-the-shelf real-time correction of multipath interference in time-of-flight imaging , 2017, ACM Trans. Graph..

[13]  Rahul Nair,et al.  Simulation of Time-of-Flight Sensors using Global Illumination , 2013, VMV.

[14]  Nikos Komodakis,et al.  Learning to Detect Ground Control Points for Improving the Accuracy of Stereo Matching , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Stefano Mattoccia,et al.  Locally Consistent ToF and Stereo Data Fusion , 2012, ECCV Workshops.

[16]  Guido M. Cortelazzo,et al.  A Probabilistic Approach to ToF and Stereo Data Fusion , 2010 .

[17]  Pierluigi Salvo Rossi,et al.  Channel-Aware Decision Fusion in Distributed MIMO Wireless Sensor Networks: Decode-and-Fuse vs. Decode-then-Fuse , 2012, IEEE Transactions on Wireless Communications.

[18]  Stefano Mattoccia,et al.  Learning from scratch a confidence measure , 2016, BMVC.

[19]  Stefano Mattoccia,et al.  Learning a General-Purpose Confidence Measure Based on O(1) Features and a Smarter Aggregation Strategy for Semi Global Matching , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[20]  Guido M. Cortelazzo,et al.  Probabilistic ToF and Stereo Data Fusion Based on Mixed Pixels Measurement Models , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  T. Kahlmann,et al.  Calibration and development for increased accuracy of 3D range imaging cameras , 2008 .

[22]  Fulvio Rinaudo,et al.  SR-4000 and CamCube3.0 Time of Flight (ToF) Cameras: Tests and Comparison , 2012, Remote. Sens..

[23]  Fabio Remondino,et al.  TOF Range-Imaging Cameras , 2013 .

[24]  Matthew D. Zeiler.  ADADELTA: An Adaptive Learning Rate Method , 2012, ArXiv.

[25]  Xiuping Jia,et al.  Deep Fusion of Remote Sensing Data for Accurate Classification , 2017, IEEE Geoscience and Remote Sensing Letters.

[26]  Radu Horaud,et al.  Fusion of Range and Stereo Data for High-Resolution Scene-Modeling , 2015, IEEE Trans. Pattern Anal. Mach. Intell..

[27]  Ruigang Yang,et al.  Spatial-Depth Super Resolution for Range Images , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[28]  Dah-Jye Lee,et al.  Review of stereo vision algorithms and their suitability for resource-limited systems , 2013, Journal of Real-Time Image Processing.

[29]  Kuk-Jin Yoon,et al.  Leveraging stereo matching with learning-based confidence measures , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Gianluca Agresti,et al.  Combination of Spatially-Modulated ToF and Structured Light for MPI-Free Depth Estimation , 2018, ECCV Workshops.

[31]  Marcus A. Magnor,et al.  A Survey on Time-of-Flight Stereo Fusion , 2013, Time-of-Flight and Depth Imaging.

[32]  Gianluca Agresti,et al.  Deep Learning for Multi-path Error Removal in ToF Sensors , 2018, ECCV Workshops.

[33]  Mirko Schmidt,et al.  SRA: Fast Removal of General Multipath for ToF Sensors , 2014, ECCV.

[34]  Stefano Mattoccia,et al.  Deep Stereo Fusion: Combining Multiple Disparity Hypotheses with Deep-Learning , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[35]  Radu Horaud,et al.  Time-of-Flight Cameras: Principles, Methods and Applications , 2012 .

[36]  Heiko Hirschmüller,et al.  Stereo Processing by Semiglobal Matching and Mutual Information , 2008, IEEE Trans. Pattern Anal. Mach. Intell..

[37]  Björn E. Ottersten,et al.  Pixel weighted average strategy for depth sensor data fusion , 2010, 2010 IEEE International Conference on Image Processing.

[38]  Horst Bischof,et al.  OctNetFusion: Learning Depth Fusion from Data , 2017, 2017 International Conference on 3D Vision (3DV).

[39]  Rahul Nair,et al.  Ensemble Learning for Confidence Measures in Stereo Vision , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[40]  Xiaoyan Hu,et al.  A Quantitative Evaluation of Confidence Measures for Stereo Vision , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  Ruigang Yang,et al.  Fusion of time-of-flight depth and stereo for high accuracy depth maps , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[42]  Stefano Mattoccia,et al.  A locally global approach to stereo correspondence , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[43]  Ruigang Yang,et al.  Reliability Fusion of Time-of-Flight Depth and Stereo Geometry for High Quality Depth Maps , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[44]  Rahul Nair,et al.  High Accuracy TOF and Stereo Sensor Fusion at Interactive Rates , 2012, ECCV Workshops.

[45]  Antonio Pescapè,et al.  Multi-classification approaches for classifying mobile app traffic , 2018, J. Netw. Comput. Appl..

[46]  Piergiorgio Sartor,et al.  Unsupervised Domain Adaptation for ToF Data Denoising With Adversarial Learning , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[47]  Ruigang Yang,et al.  Spatial-Temporal Fusion for High Accuracy Depth Maps Using Dynamic MRFs , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.