A Survey on Time-of-Flight Stereo Fusion

Due to the demand for depth maps of higher quality than possible with a single depth imaging technique today, there has been an increasing interest in the combination of different depth sensors to produce a “super-camera” that is more than the sum of the individual parts. In this survey paper, we give an overview over methods for the fusion of Time-of-Flight (ToF) and passive stereo data as well as applications of the resulting high quality depth maps. Additionally, we provide a tutorial-based introduction to the principles behind ToF stereo fusion and the evaluation criteria used to benchmark these methods.

[1]  Leonard McMillan,et al.  Image-based rendering: A new interface between computer vision and computer graphics , 1999, COMG.

[2]  J. Nocedal.  Updating Quasi-Newton Matrices With Limited Storage , 1980 .

[3]  Heiko Hirschmüller,et al.  Evaluation of Cost Functions for Stereo Matching , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  Young Min Kim,et al.  Multi-view image and ToF sensor fusion for dense 3D reconstruction , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[5]  Heiko Hirschmüller,et al.  Stereo Processing by Semiglobal Matching and Mutual Information , 2008, IEEE Trans. Pattern Anal. Mach. Intell..

[6]  M. Pollefeys,et al.  A Unified Approach to Calibrate a Network of Camcorders and ToF cameras , 2008 .

[7]  Alexander Verl,et al.  Combination of Time-of-Flight depth and stereo using semiglobal optimization , 2011, 2011 IEEE International Conference on Robotics and Automation.

[8]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[9]  Jiawen Chen,et al.  The video mesh: A data structure for image-based three-dimensional video editing , 2011, 2011 IEEE International Conference on Computational Photography (ICCP).

[10]  Gabriel Taubin,et al.  Stereoscopic Cinema , 2015, Image and Geometry Processing for 3-D Cinematography.

[11]  Paul A. Beardsley,et al.  Stereoscopic Cinema , 2010, Image and Geometry Processing for 3-D Cinematography.

[12]  Berthold K. P. Horn,et al.  Closed-form solution of absolute orientation using unit quaternions , 1987 .

[13]  Bernd Jähne,et al.  Ground Truth for Evaluating Time of Flight Imaging , 2013, Time-of-Flight and Depth Imaging.

[14]  Jan-Michael Frahm,et al.  Real-Time Plane-Sweeping Stereo with Multiple Sweeping Directions , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Ruigang Yang,et al.  Spatial-Temporal Fusion for High Accuracy Depth Maps Using Dynamic MRFs , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Hans-Peter Seidel,et al.  Real-time lens blur effects and focus control , 2010, SIGGRAPH 2010.

[17]  Mario Fritz,et al.  Improving the Kinect by Cross-Modal Stereo , 2011, BMVC.

[18]  Reinhard Koch,et al.  A Combined Approach for Estimating Patchlets from PMD Depth Images and Stereo Intensity Images , 2007, DAGM-Symposium.

[19]  Marcus A. Magnor,et al.  Integrating approximate depth data into dense image correspondence estimation , 2012, CVMP.

[20]  Rasmus Larsen,et al.  Fusion of stereo vision and Time-Of-Flight imaging for improved 3D estimation , 2008, Int. J. Intell. Syst. Technol. Appl..

[21]  Olga Veksler,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[22]  Vladimir Kolmogorov,et al.  Computing visual correspondence with occlusions using graph cuts , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[23]  Jiří Matas,et al.  Computer Vision - ECCV 2004 , 2004, Lecture Notes in Computer Science.

[24]  Young Min Kim,et al.  Design and calibration of a multi-view TOF sensor fusion system , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[25]  Marc Alexa,et al.  Combining Time-Of-Flight depth and stereo images without accurate extrinsic calibration , 2008, Int. J. Intell. Syst. Technol. Appl..

[26]  Andrew Zisserman,et al.  Multiple View Geometry , 1999 .

[27]  Nanning Zheng,et al.  Stereo Matching Using Belief Propagation , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  Alexander Toet,et al.  Merging thermal and visual images by a contrast pyramid , 1989 .

[29]  Matthias Zwicker,et al.  Stereoscopic 3D copy & paste , 2010, SIGGRAPH 2010.

[30]  Ruigang Yang,et al.  Reliability Fusion of Time-of-Flight Depth and Stereo Geometry for High Quality Depth Maps , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Guido M. Cortelazzo,et al.  A Probabilistic Approach to ToF and Stereo Data Fusion , 2010 .

[32]  Rahul Nair,et al.  High Accuracy TOF and Stereo Sensor Fusion at Interactive Rates , 2012, ECCV Workshops.

[33]  Klaus-Dieter Kuhnert,et al.  Fusion ofStereo-Camera andPMD-Camera DataforReal- TimeSuited Precise 3DEnvironment Reconstruction , 2006 .

[34]  Mei Yu,et al.  A Color Correction Algorithm of Multi-view Video Based on Depth Segmentation , 2008, 2008 International Symposium on Computer Science and Computational Technology.

[35]  Matthieu Guillaumin,et al.  Segmentation Propagation in ImageNet , 2012, ECCV.

[36]  B. Huhle,et al.  Integrating 3D Time-of-Flight Camera Data and High Resolution Images for 3DTV Applications , 2007, 2007 3DTV Conference.

[37]  Reinhard Koch,et al.  Technical Foundation and Calibration Methods for Time-of-Flight Cameras , 2013, Time-of-Flight and Depth Imaging.

[38]  Stephan Meister,et al.  Erratum: High Accuracy TOF and Stereo Sensor Fusion at Interactive Rates , 2012, ECCV 2012.

[39]  Tim Weyrich,et al.  Capturing Time-of-Flight data with confidence , 2011, CVPR 2011.

[40]  Rahul Nair,et al.  Denoising Strategies for Time-of-Flight Data , 2013, Time-of-Flight and Depth Imaging.

[41]  Radu Horaud,et al.  High-resolution depth maps based on TOF-stereo fusion , 2012, 2012 IEEE International Conference on Robotics and Automation.

[42]  Chris A. Glasbey,et al.  Combining Stereo and Time-of-Flight Images with Application to Automatic Plant Phenotyping , 2011, SCIA.

[43]  John G. Apostolopoulos,et al.  Fusion of active and passive sensors for fast 3D capture , 2010, 2010 IEEE International Workshop on Multimedia Signal Processing.

[44]  Michael S. Brown,et al.  High quality depth map upsampling for 3D-TOF cameras , 2011, 2011 International Conference on Computer Vision.

[45]  Jitendra Malik,et al.  Shape, albedo, and illumination from a single image of an unknown object , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[46]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[47]  Reinhard Koch,et al.  Dynamic 3D Imaging, DAGM 2009 Workshop, Dyn3D 2009, Jena, Germany, September 9, 2009. Proceedings , 2009, Dyn3D.

[48]  VekslerOlga,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001 .

[49]  Joachim Hornegger,et al.  Standardization of intensity-values acquired by Time-of-Flight-cameras , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[50]  Leszek Wojnar,et al.  Image Analysis , 1998 .

[51]  Horst Bischof,et al.  A Duality Based Approach for Realtime TV-L1 Optical Flow , 2007, DAGM-Symposium.

[52]  Ruigang Yang,et al.  Fusion of time-of-flight depth and stereo for high accuracy depth maps , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[53]  Stefano Mattoccia,et al.  A locally global approach to stereo correspondence , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[54]  Jean-Yves Bouguet,et al.  Camera calibration toolbox for matlab , 2001 .

[55]  Frédo Durand,et al.  Flash photography enhancement via intrinsic relighting , 2004, SIGGRAPH 2004.

[56]  Josip Krapac,et al.  Infrared-Visual Image Registration Based on Corners and Hausdorff Distance , 2007, SCIA.

[57]  Nassir Navab,et al.  Stereo time-of-flight , 2011, 2011 International Conference on Computer Vision.

[58]  S. Birchfiled.  A Pixel Dissimilarity Measure That Is Insensitive to Image Sampling , 1998 .

[59]  Stefano Mattoccia,et al.  Locally Consistent ToF and Stereo Data Fusion , 2012, ECCV Workshops.

[60]  Andrew J. Davison,et al.  DTAM: Dense tracking and mapping in real-time , 2011, 2011 International Conference on Computer Vision.

[61]  Reinhard Koch,et al.  Dense Depth Maps from Low Resolution Time-of-Flight Depth and High Resolution Color Views , 2009, ISVC.

[62]  Maarten Vergauwen,et al.  A Hierarchical Symmetric Stereo Algorithm Using Dynamic Programming , 2002, International Journal of Computer Vision.

[63]  R. Koch,et al.  CALIBRATION OF A PMD-CAMERA USING A PLANAR CALIBRATION PATTERN TOGETHER WITH A MULTI-CAMERA SETUP , 2008 .

[64]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[65]  Li Zhang,et al.  Spacetime stereo: shape recovery for dynamic scenes , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[66]  Thomas Brox,et al.  High Accuracy Optical Flow Estimation Based on a Theory for Warping , 2004, ECCV.

[67]  Radim Sára,et al.  Efficient Sampling of Disparity Space for Fast And Accurate Matching , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[68]  Michael F. Cohen,et al.  Digital photography with flash and no-flash image pairs , 2004, ACM Trans. Graph..

[69]  Guillaume-Alexandre Bilodeau,et al.  Visible and infrared image registration using trajectories and composite foreground images , 2011, Image Vis. Comput..

[70]  Hans-Peter Seidel,et al.  Highlight microdisparity for improved gloss depiction , 2012, ACM Trans. Graph..

[71]  David A. McAllester,et al.  Particle Belief Propagation , 2009, AISTATS.

[72]  Marc Alexa,et al.  Depth Imaging by Combining Time-of-Flight and On-Demand Stereo , 2009, Dyn3D.

[73]  Berthold K. P. Horn,et al.  Shape from shading , 1989 .