Image-guided ToF depth upsampling: a survey

Recently, there has been remarkable growth of interest in the development and applications of time-of-flight (ToF) depth cameras. Despite the permanent improvement of their characteristics, the practical applicability of ToF cameras is still limited by low resolution and quality of depth measurements. This has motivated many researchers to combine ToF cameras with other sensors in order to enhance and upsample depth images. In this paper, we review the approaches that couple ToF depth images with high-resolution optical images. Other classes of upsampling methods are also briefly discussed. Finally, we provide an overview of performance evaluation tests presented in the related studies.

[1]  Aggelos K. Katsaggelos,et al.  Upsampling and denoising of depth maps via joint-segmentation , 2012, 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO).

[2]  Narendra Ahuja,et al.  Real-time O(1) bilateral filtering , 2009, CVPR.

[3]  Gang Wang,et al.  Fusion of Median and Bilateral Filtering for Range Image Upsampling , 2013, IEEE Transactions on Image Processing.

[4]  Reinhard Koch,et al.  Pose estimation and map building with a Time-Of-Flight-camera for robot navigation , 2008, Int. J. Intell. Syst. Technol. Appl..

[5]  Xia Liu,et al.  Hand gesture recognition using depth data , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[6]  Shahram Izadi,et al.  Modeling Kinect Sensor Noise for Improved 3D Reconstruction and Tracking , 2012, 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission.

[7]  Ruigang Yang,et al.  Spatial-Depth Super Resolution for Range Images , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Horst Bischof,et al.  Image Guided Depth Upsampling Using Anisotropic Total Generalized Variation , 2013, 2013 IEEE International Conference on Computer Vision.

[9]  Jian Sun,et al.  Guided Image Filtering , 2010, ECCV.

[10]  Roland Siegwart,et al.  A state-of-the-art 3D sensor for robot navigation , 2004 .

[11]  Kai-Lung Hua,et al.  Edge-Preserving Depth Map Upsampling by Joint Trilateral Filter , 2018, IEEE Transactions on Cybernetics.

[12]  D. Holz,et al.  3 D Pose Estimation and Mapping with Time-of-Flight Cameras , 2008 .

[13]  Marc Alexa,et al.  Combining Time-Of-Flight depth and stereo images without accurate extrinsic calibration , 2008, Int. J. Intell. Syst. Technol. Appl..

[14]  Carme Torras,et al.  Object modeling using a ToF camera under an uncertainty reduction approach , 2010, 2010 IEEE International Conference on Robotics and Automation.

[15]  Gerard de Haan,et al.  Evaluation of efficient high quality depth upsampling methods for 3DTV , 2013, Electronic Imaging.

[16]  Radim Sára,et al.  Efficient Sampling of Disparity Space for Fast And Accurate Matching , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Björn E. Ottersten,et al.  Pixel weighted average strategy for depth sensor data fusion , 2010, 2010 IEEE International Conference on Image Processing.

[18]  Sebastian Schwarz,et al.  A Weighted Optimization Approach to Time-of-Flight Sensor Fusion , 2014, IEEE Transactions on Image Processing.

[19]  Minh N. Do,et al.  Depth Video Enhancement Based on Weighted Mode Filtering , 2012, IEEE Transactions on Image Processing.

[20]  Peyman Milanfar,et al.  A Tour of Modern Image Filtering: New Insights and Methods, Both Practical and Theoretical , 2013, IEEE Signal Processing Magazine.

[21]  Gerard de Haan,et al.  Overview of efficient high-quality state-of-the-art depth enhancement methods by thorough design space exploration , 2015, Journal of Real-Time Image Processing.

[22]  Zhengyou Zhang,et al.  Microsoft Kinect Sensor and Its Effect , 2012, IEEE Multim..

[23]  Paul Newman,et al.  Image and Sparse Laser Fusion for Dense Scene Reconstruction , 2009, FSR.

[24]  Mark E. Campbell,et al.  Segmentation of dense range information in complex urban scenes , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[25]  Ruigang Yang,et al.  A Performance Study on Different Cost Aggregation Approaches Used in Real-Time Stereo Matching , 2007, International Journal of Computer Vision.

[26]  Jean-Michel Morel,et al.  A Review of Image Denoising Algorithms, with a New One , 2005, Multiscale Model. Simul..

[27]  Diego F. Nehab,et al.  Efficiently combining positions and normals for precise 3D geometry , 2005, SIGGRAPH 2005.

[28]  Feng Liu,et al.  Depth Enhancement via Low-Rank Matrix Completion , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[29]  Chang-Yeong Kim,et al.  Discrete and continuous optimizations for depth image super-resolution , 2012, Electronic Imaging.

[30]  Sven Wachsmuth,et al.  Laser-based navigation enhanced with 3D time-of-flight data , 2009, 2009 IEEE International Conference on Robotics and Automation.

[31]  Kiriakos N. Kutulakos,et al.  Transparent and Specular Object Reconstruction , 2010, Comput. Graph. Forum.

[32]  Ventseslav Sainov,et al.  3-D Time-Varying Scene Capture Technologies—A Survey , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[33]  Gang Yang,et al.  Depth super resolution using bilateral filter , 2011, 2011 4th International Congress on Image and Signal Processing.

[34]  Rasmus Larsen,et al.  Fusion of stereo vision and Time-Of-Flight imaging for improved 3D estimation , 2008, Int. J. Intell. Syst. Technol. Appl..

[35]  Dani Lischinski,et al.  Joint bilateral upsampling , 2007, SIGGRAPH 2007.

[36]  S. Foix,et al.  Lock-in Time-of-Flight (ToF) Cameras: A Survey , 2011, IEEE Sensors Journal.

[37]  Hongbin Zha,et al.  A Bayesian Approach to Uncertainty-Based Depth Map Super Resolution , 2012, ACCV.

[38]  Alfred M. Bruckstein,et al.  RGBD-fusion: Real-time high precision depth recovery , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Ruigang Yang,et al.  Fusion of time-of-flight depth and stereo for high accuracy depth maps , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[40]  Yao Zhao,et al.  Depth map up-sampling with fractal dimension and texture-depth boundary consistencies , 2017, Neurocomputing.

[41]  Rüdiger Dillmann,et al.  Sensor fusion for 3D human body tracking with an articulated 3D body model , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[42]  José García Rodríguez,et al.  A Quantitative Comparison of Calibration Methods for RGB-D Sensors Using Different Technologies , 2017, Sensors.

[43]  Yong Dou,et al.  Depth enhancement via non-local means filter , 2015, 2015 Seventh International Conference on Advanced Computational Intelligence (ICACI).

[44]  Derek D. Lichti,et al.  3D Cameras: Errors, Calibration and Orientation , 2013 .

[45]  Young Min Kim,et al.  Multi-view image and ToF sensor fusion for dense 3D reconstruction , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[46]  Zhigeng Pan,et al.  Real-Time Spatial and Depth Upsampling for Range Data , 2011, Trans. Comput. Sci..

[47]  Marc Alexa,et al.  Depth Imaging by Combining Time-of-Flight and On-Demand Stereo , 2009, Dyn3D.

[48]  Sebastian Thrun,et al.  LidarBoost: Depth superresolution for ToF 3D shape scanning , 2009, CVPR.

[49]  Cheng Wang,et al.  Automated Road Information Extraction From Mobile Laser Scanning Data , 2015, IEEE Transactions on Intelligent Transportation Systems.

[50]  Horst Bischof,et al.  ATGV-Net: Accurate Depth Super-Resolution , 2016, ECCV.

[51]  Hans-Peter Seidel,et al.  Coherent Spatiotemporal Filtering, Upsampling and Rendering of RGBZ Videos , 2012, Comput. Graph. Forum.

[52]  Luigi di Stefano,et al.  People Tracking Using a Time-of-Flight Depth Sensor , 2006, 2006 IEEE International Conference on Video and Signal Based Surveillance.

[53]  Yo-Sung Ho,et al.  High-quality multi-view depth generation using multiple color and depth cameras , 2010, 2010 IEEE International Conference on Multimedia and Expo.

[54]  Sebastian Schwarz,et al.  Temporal consistent depth map upscaling for 3DTV , 2014, Electronic Imaging.

[55]  John G. Apostolopoulos,et al.  Fusion of active and passive sensors for fast 3D capture , 2010, 2010 IEEE International Workshop on Multimedia Signal Processing.

[56]  Ruigang Yang,et al.  Spatial-Temporal Fusion for High Accuracy Depth Maps Using Dynamic MRFs , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[57]  Milan Sonka,et al.  Image Processing, Analysis and Machine Vision , 1993, Springer US.

[58]  Larry Li,et al.  Time-of-Flight Camera - An Introduction , 2014 .

[59]  Mirko Schmidt.  Analysis, Modeling and Dynamic Optimization of 3D Time-of-Flight Imaging Systems , 2011 .

[60]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[61]  Fabio Remondino,et al.  TOF Range-Imaging Cameras , 2013 .

[62]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[63]  Danny Barash,et al.  A Fundamental Relationship between Bilateral Filtering, Adaptive Smoothing, and the Nonlinear Diffusion Equation , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[64]  Dmitry Chetverikov,et al.  Novel methods for image-guided ToF depth upsampling , 2016, 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[65]  Kikuo Fujimura,et al.  Visual Tracking Using Depth Data , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[66]  Andreas Nüchter,et al.  Robust 3D-mapping with time-of-flight cameras , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[67]  Klaus-Dieter Kuhnert,et al.  Fusion of Stereo-Camera and PMD-Camera Data for Real-Time Suited Precise 3D Environment Reconstruction , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[68]  Reinhard Koch,et al.  Time-of-Flight sensor calibration for accurate range sensing , 2010, Comput. Vis. Image Underst..

[69]  Jens T. Thielemann,et al.  Pipeline landmark detection for autonomous robot navigation using time-of-flight imagery , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[70]  Kai-Kuang Ma,et al.  A survey on super-resolution imaging , 2011, Signal Image Video Process..

[71]  Chandra Shaker Balure,et al.  Depth Image Super-Resolution: A Review and Wavelet Perspective , 2016, CVIP.

[72]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[73]  Bernd Jähne,et al.  Ground Truth for Evaluating Time of Flight Imaging , 2013, Time-of-Flight and Depth Imaging.

[74]  Reinhard Koch,et al.  Time-of-Flight and Depth Imaging. Sensors, Algorithms, and Applications , 2013, Lecture Notes in Computer Science.

[75]  A. K. Riemens,et al.  Multistep joint bilateral depth upsampling , 2009, Electronic Imaging.

[76]  In-So Kweon,et al.  High Quality Shape from a Single RGB-D Image under Uncalibrated Natural Illumination , 2013, 2013 IEEE International Conference on Computer Vision.

[77]  Marc Alexa,et al.  Exposure Fusion for Time‐Of‐Flight Imaging , 2011, Comput. Graph. Forum.

[78]  Daniel Herrera C,et al.  Joint depth and color camera calibration with distortion correction. , 2012, IEEE transactions on pattern analysis and machine intelligence.

[79]  José-Raúl Ruiz-Sarmiento,et al.  Improving Human Face Detection through TOF Cameras for Ambient Intelligence Applications , 2011, ISAmI.

[80]  Xin Jin,et al.  Enhanced Joint Trilateral Up-sampling for Super-Resolution , 2016, PCM.

[81]  Reinhard Koch,et al.  Technical Foundation and Calibration Methods for Time-of-Flight Cameras , 2013, Time-of-Flight and Depth Imaging.

[82]  Carme Torras,et al.  Exploitation of time-of-flight (ToF) cameras , 2010 .

[83]  Kevin P. Murphy,et al.  Machine learning - a probabilistic perspective , 2012, Adaptive computation and machine learning series.

[84]  Joaquim Salvi,et al.  A state of the art in structured light patterns for surface profilometry , 2010, Pattern Recognit..

[85]  Carlo Tomasi,et al.  3D head tracking based on recognition and interpolation using a time-of-flight depth sensor , 2004, CVPR 2004.

[86]  Sabine Süsstrunk,et al.  Color image dehazing using the near-infrared , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[87]  Carme Torras,et al.  3D Object Reconstruction from Swissranger Sensor Data using a Spring-mass Model , 2009, VISAPP.

[88]  Markus Vincze,et al.  Selecting good corners for structure and motion recovery using a time-of-flight camera , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[89]  Lai-Man Po,et al.  Depth map misalignment correction and dilation for DIBR view synthesis , 2013, Signal Process. Image Commun..

[90]  Akira Kojima,et al.  Self-similarity matching with predictive linear upsampling for depth map , 2016, 2016 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[91]  Rasmus Larsen,et al.  TOF imaging in Smart room environments towards improved people tracking , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[92]  Suyash P. Awate,et al.  Higher-order image statistics for unsupervised, information-theoretic, adaptive, image filtering , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[93]  Pierre Kornprobst,et al.  Bilateral Filtering , 2009 .

[94]  Reinhard Koch,et al.  A Comparison of PMD-Cameras and Stereo-Vision for the Task of Surface Reconstruction using Patchlets , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[95]  Guy Gilboa,et al.  A Depth Restoration Occlusionless Temporal Dataset , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[96]  Andrew W. Fitzgibbon,et al.  KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera , 2011, UIST.

[97]  Sebastian Schwarz,et al.  Depth Sensing for 3DTV: A Survey , 2013, IEEE MultiMedia.

[98]  Timo Schairer,et al.  Fusion of range and color images for denoising and resolution enhancement with a non-local filter , 2010, Comput. Vis. Image Underst..

[99]  Weijia Zhou,et al.  Depth map super-resolution via extended weighted mode filtering , 2016, 2016 Visual Communications and Image Processing (VCIP).

[100]  Narendra Ahuja,et al.  Deep Joint Image Filtering , 2016, ECCV.

[101]  Michael S. Brown,et al.  High quality depth map upsampling for 3D-TOF cameras , 2011, 2011 International Conference on Computer Vision.

[102]  Xiaoou Tang,et al.  Depth Map Super-Resolution by Deep Multi-Scale Guidance , 2016, ECCV.

[103]  Sebastian Thrun,et al.  An Application of Markov Random Fields to Range Sensing , 2005, NIPS.

[104]  Roberto Manduchi,et al.  Bilateral filtering for gray and color images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[105]  S. Hussmann,et al.  Robot Vision System based on a 3D-TOF Camera , 2007, 2007 IEEE Instrumentation & Measurement Technology Conference IMTC 2007.

[106]  Kwanghoon Sohn,et al.  Spatial and temporal up-conversion technique for depth video , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[107]  Yun-Suk Kang.  Multi-View Depth Generation using Multi-Depth Camera System , 2010 .

[108]  Ulugbek Kamilov,et al.  Depth superresolution using motion adaptive regularization , 2016, 2016 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[109]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[110]  Xi Wang,et al.  High-Resolution Stereo Datasets with Subpixel-Accurate Ground Truth , 2014, GCPR.

[111]  Hichem Sahli,et al.  Outdoor Terrain Traversability Analysis for Robot Navigation using a Time-Of-Flight Camera , 2011 .

[112]  Gabriel J. Brostow,et al.  Patch Based Synthesis for Single Depth Image Super-Resolution , 2012, ECCV.

[113]  Reinhard Koch,et al.  Dense Depth Maps from Low Resolution Time-of-Flight Depth and High Resolution Color Views , 2009, ISVC.

[114]  Moncef Gabbouj,et al.  Weighted median filters: a tutorial , 1996 .

[115]  Srimanta Mandal,et al.  Depth Map Restoration From Undersampled Data , 2017, IEEE Transactions on Image Processing.

[116]  Richard Szeliski,et al.  Computer Vision - Algorithms and Applications , 2011, Texts in Computer Science.

[117]  Christian Wöhler,et al.  An introduction to image-based 3D surface reconstruction and a survey of photometric stereo methods , 2011 .

[118]  Dieter Fox,et al.  A large-scale hierarchical multi-view RGB-D object dataset , 2011, 2011 IEEE International Conference on Robotics and Automation.

[119]  Michael F. Cohen,et al.  Stylizing 2.5-D video , 2006, NPAR.

[120]  Stefan May,et al.  Calibration and registration for precise surface reconstruction with Time-Of-Flight cameras , 2008, Int. J. Intell. Syst. Technol. Appl..

[121]  O. Loffeld,et al.  Comparison of Depth Super-Resolution Methods for 2 D / 3 D Images , 2011 .

[122]  Raanan Fattal,et al.  Image upsampling via imposed edge statistics , 2007, ACM Trans. Graph..

[123]  Carme Torras,et al.  3D modelling of leaves from color and ToF data for robotized plant measuring , 2011, 2011 IEEE International Conference on Robotics and Automation.

[124]  Karl Kunisch,et al.  Total Generalized Variation , 2010, SIAM J. Imaging Sci..

[125]  Sebastian Thrun,et al.  3D shape scanning with a time-of-flight camera , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[126]  In-So Kweon,et al.  Time-of-Flight Sensor Calibration for a Color and Depth Camera Pair , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[127]  Bodo Rosenhahn,et al.  On Calibration of a Low-Cost Time-of-Flight Camera , 2014, ECCV Workshops.

[128]  Minh N. Do,et al.  A revisit to MRF-based depth map super-resolution and enhancement , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[129]  Marcus A. Magnor,et al.  A Survey on Time-of-Flight Stereo Fusion , 2013, Time-of-Flight and Depth Imaging.

[130]  Sebastian Schwarz,et al.  Multivariate Sensitivity Analysis of Time-of-Flight Sensor Fusion , 2014 .

[131]  Fatih Porikli,et al.  Constant time O(1) bilateral filtering , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[132]  Sang Uk Lee,et al.  Superpixel-based depth image super-resolution , 2012, Electronic Imaging.

[133]  Reinhard Koch,et al.  Time‐of‐Flight Cameras in Computer Graphics , 2010, Comput. Graph. Forum.

[134]  Radu Horaud,et al.  Time-of-Flight Cameras , 2012, SpringerBriefs in Computer Science.

[135]  Narendra Ahuja,et al.  Real-time O(1) bilateral filtering , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[136]  Ruigang Yang,et al.  Reliability Fusion of Time-of-Flight Depth and Stereo Geometry for High Quality Depth Maps , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[137]  D. Falie,et al.  Wide range Time of Flight camera for outdoor surveillance , 2008, 2008 Microwaves, Radar and Remote Sensing Symposium.

[138]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[139]  Sebastian Thrun,et al.  A Noise‐aware Filter for Real‐time Depth Upsampling , 2008 .

[140]  Stephen Lin,et al.  Shading-Based Shape Refinement of RGB-D Images , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[141]  Sebastian Schwarz.  Gaining Depth : Time-of-Flight Sensor Fusion for Three-Dimensional Video Content Creation , 2014 .

[142]  Markus Vincze,et al.  Experimental Evaluation of State of the Art 3 D-Sensors for Mobile Robot Navigation 1 ) , 2007 .

[143]  Sebastian Thrun,et al.  Upsampling range data in dynamic environments , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[144]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[145]  Michael S. Brown,et al.  High-Quality Depth Map Upsampling and Completion for RGB-D Cameras , 2014, IEEE Transactions on Image Processing.

[146]  David Fofi,et al.  A comparative survey on invisible structured light , 2004, IS&T/SPIE Electronic Imaging.

[147]  Carsten Rother,et al.  Depth Super Resolution by Rigid Body Self-Similarity in 3D , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[148]  Hongbin Zha,et al.  A Joint Learning-Based Method for Multi-view Depth Map Super Resolution , 2013, 2013 2nd IAPR Asian Conference on Pattern Recognition.

[149]  J. D. van Ouwerkerk,et al.  Image super-resolution survey , 2006, Image Vis. Comput..