Probabilistic ToF and Stereo Data Fusion Based on Mixed Pixels Measurement Models

This paper proposes a method for fusing data acquired by a ToF camera and a stereo pair based on a model for depth measurement by ToF cameras which accounts also for depth discontinuity artifacts due to the mixed pixel effect. Such model is exploited within both a ML and a MAP-MRF frameworks for ToF and stereo data fusion. The proposed MAP-MRF framework is characterized by site-dependent range values, a rather important feature since it can be used both to improve the accuracy and to decrease the computational complexity of standard MAP-MRF approaches. This paper, in order to optimize the site dependent global cost function characteristic of the proposed MAP-MRF approach, also introduces an extension to Loopy Belief Propagation which can be used in other contexts. Experimental data validate the proposed ToF measurements model and the effectiveness of the proposed fusion techniques.

[1]  Fulvio Rinaudo,et al.  SR-4000 and CamCube3.0 Time of Flight (ToF) Cameras: Tests and Comparison , 2012, Remote. Sens..

[2]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[3]  Ruigang Yang,et al.  Reliability Fusion of Time-of-Flight Depth and Stereo Geometry for High Quality Depth Maps , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Rahul Nair,et al.  High Accuracy TOF and Stereo Sensor Fusion at Interactive Rates , 2012, ECCV Workshops.

[5]  Young Min Kim,et al.  Multi-view image and ToF sensor fusion for dense 3D reconstruction , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[6]  Guido M. Cortelazzo,et al.  Time-of-Flight Cameras and Microsoft Kinect™ , 2012, Springer Briefs in Electrical and Computer Engineering.

[7]  Vladimir Kolmogorov,et al.  An experimental comparison of min-cut/max- flow algorithms for energy minimization in vision , 2001, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Martin J. Wainwright,et al.  MAP estimation via agreement on (hyper)trees: Message-passing and linear programming , 2005, ArXiv.

[9]  Heiko Hirschmüller,et al.  Stereo Processing by Semiglobal Matching and Mutual Information , 2008, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  Mirko Schmidt.  Analysis, Modeling and Dynamic Optimization of 3D Time-of-Flight Imaging Systems , 2011 .

[11]  Robert Mahony,et al.  Statistical analysis of measurement processes for time-of-flight cameras , 2009, Optical Engineering + Applications.

[12]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[13]  Fabio Remondino,et al.  TOF Range-Imaging Cameras , 2013 .

[14]  Sebastian Thrun,et al.  Upsampling range data in dynamic environments , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[15]  Li Zhang,et al.  Spacetime stereo: shape recovery for dynamic scenes , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[16]  William T. Freeman,et al.  Comparison of graph cuts with belief propagation for stereo, using identical MRF parameters , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[17]  R. Raskar,et al.  Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging , 2012, Nature Communications.

[18]  Gang Wang,et al.  Fusion of Median and Bilateral Filtering for Range Image Upsampling , 2013, IEEE Transactions on Image Processing.

[19]  Ruigang Yang,et al.  Spatial-Depth Super Resolution for Range Images , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  T. Kahlmann,et al.  Calibration and development for increased accuracy of 3D range imaging cameras , 2008 .

[21]  M. Cree,et al.  Mixed pixel return separation for a full-field ranger , 2008, 2008 23rd International Conference Image and Vision Computing New Zealand.

[22]  Guido M. Cortelazzo,et al.  A Probabilistic Approach to ToF and Stereo Data Fusion , 2010 .

[23]  Qionghai Dai,et al.  Decomposing Global Light Transport Using Time of Flight Imaging , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Pietro Zanuttigh,et al.  A Novel Interpolation Scheme for Range Data with Side Information , 2009, 2009 Conference for Visual Media Production.

[25]  Rasmus Larsen,et al.  Fusion of stereo vision and Time-Of-Flight imaging for improved 3D estimation , 2008, Int. J. Intell. Syst. Technol. Appl..

[26]  Szymon Rusinkiewicz,et al.  Spacetime stereo: a unifying framework for depth from triangulation , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Stefano Mattoccia,et al.  Locally Consistent ToF and Stereo Data Fusion , 2012, ECCV Workshops.

[28]  Guido M. Cortelazzo,et al.  Edge-preserving interpolation of depth data exploiting color information , 2013, Ann. des Télécommunications.

[29]  Richard Szeliski,et al.  A Comparative Study of Energy Minimization Methods for Markov Random Fields with Smoothness-Based Priors , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Radu Horaud,et al.  Time-of-Flight Cameras , 2012, SpringerBriefs in Computer Science.

[31]  Stefano Mattoccia,et al.  Accurate and Efficient Cost Aggregation Strategy for Stereo Correspondence Based on Approximated Joint Bilateral Filtering , 2009, ACCV.

[32]  Robert Lange,et al.  3D time-of-flight distance measurement with custom solid-state image sensors in CMOS/CCD-technology , 2006 .

[33]  Marcus A. Magnor,et al.  A Survey on Time-of-Flight Stereo Fusion , 2013, Time-of-Flight and Depth Imaging.

[34]  Radu Horaud,et al.  Time-of-Flight Cameras: Principles, Methods and Applications , 2012 .

[35]  John G. Apostolopoulos,et al.  Fusion of active and passive sensors for fast 3D capture , 2010, 2010 IEEE International Workshop on Multimedia Signal Processing.

[36]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[37]  H. Hirschmüller.  Ieee Transactions on Pattern Analysis and Machine Intelligence 1 Stereo Processing by Semi-global Matching and Mutual Information , 2022 .

[38]  Carsten Rother,et al.  PatchMatch Stereo - Stereo Matching with Slanted Support Windows , 2011, BMVC.

[39]  Radu Horaud,et al.  Fusion of Range and Stereo Data for High-Resolution Scene-Modeling , 2015, IEEE Trans. Pattern Anal. Mach. Intell..

[40]  Ruigang Yang,et al.  Search Space Reduction for MRF Stereo , 2008, ECCV.

[41]  Michael J. Black,et al.  On the unification of line processes, outlier rejection, and robust statistics with applications in early vision , 1996, International Journal of Computer Vision.

[42]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[43]  Marc Alexa,et al.  Depth Imaging by Combining Time-of-Flight and On-Demand Stereo , 2009, Dyn3D.

[44]  Sebastian Thrun,et al.  An Application of Markov Random Fields to Range Sensing , 2005, NIPS.

[45]  Nanning Zheng,et al.  Stereo Matching Using Belief Propagation , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[46]  Carlo Tomasi,et al.  Depth Discontinuities by Pixel-to-Pixel Stereo , 1999, International Journal of Computer Vision.

[47]  Ruigang Yang,et al.  Fusion of time-of-flight depth and stereo for high accuracy depth maps , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[48]  Tim Weyrich,et al.  Capturing Time-of-Flight data with confidence , 2011, CVPR 2011.

[49]  A. Hasman,et al.  Probabilistic reasoning in intelligent systems: Networks of plausible inference , 1991 .

[50]  Martin J. Wainwright,et al.  MAP estimation via agreement on trees: message-passing and linear programming , 2005, IEEE Transactions on Information Theory.

[51]  Jörg Krüger,et al.  Identification and correction of flying pixels in range camera data , 2008, SCCG.

[52]  A. Frick,et al.  Generation of 3D-TV LDV-content with Time-Of-Flight Camera , 2009, 2009 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[53]  Klaus-Dieter Kuhnert,et al.  Fusion of Stereo-Camera and PMD-Camera Data for Real-Time Suited Precise 3D Environment Reconstruction , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[54]  John W. Woods,et al.  Multidimensional Signal, Image and Video Processing and Coding [Book Reviews] , 2007, IEEE Signal Processing Magazine.

[55]  J. Besag.  On the Statistical Analysis of Dirty Pictures , 1986 .

[56]  Ruigang Yang,et al.  Spatial-Temporal Fusion for High Accuracy Depth Maps Using Dynamic MRFs , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.