Deep Learning for Multi-path Error Removal in ToF Sensors

The removal of Multi-Path Interference (MPI) is one of the major open challenges in depth estimation with Time-of-Flight (ToF) cameras. In this paper we propose a novel method for MPI removal and depth refinement exploiting an ad-hoc deep learning architecture working on data from a multi-frequency ToF camera. In order to estimate the MPI we use a Convolutional Neural Network (CNN) made of two sub-networks: a coarse network analyzing the global structure of the data at a lower resolution and a fine one exploiting the output of the coarse network in order to remove the MPI while preserving the small details. The critical issue of the lack of ToF data with ground truth is solved by training the CNN with synthetic information. Finally, the residual zero-mean error is removed with an adaptive bilateral filter guided from a noise model for the camera. Experimental results prove the effectiveness of the proposed approach on both synthetic and real data.

[1]  Manuel Mazo,et al.  Modeling and correction of multipath interference in time of flight cameras , 2014, Image Vis. Comput..

[2]  Sebastian Thrun,et al.  A Noise‐aware Filter for Real‐time Depth Upsampling , 2008 .

[3]  MOHIT GUPTA,et al.  Phasor Imaging , 2015, ACM Trans. Graph..

[4]  Raquel Urtasun,et al.  Efficient Deep Learning for Stereo Matching , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Mirko Schmidt,et al.  SRA: Fast Removal of General Multipath for ToF Sensors , 2014, ECCV.

[6]  Stefano Mattoccia,et al.  Reliable Fusion of ToF and Stereo Depth Driven by Confidence Measures , 2016, ECCV.

[7]  Jan Kautz,et al.  Loss Functions for Image Restoration With Neural Networks , 2017, IEEE Transactions on Computational Imaging.

[8]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[9]  Ramesh Raskar,et al.  Coded time of flight cameras , 2013, ACM Trans. Graph..

[10]  Gordon Wetzstein,et al.  Deep End-to-End Time-of-Flight Imaging , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[11]  Rob Fergus,et al.  Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[12]  Rob Fergus,et al.  Depth Map Prediction from a Single Image using a Multi-Scale Deep Network , 2014, NIPS.

[13]  Ludovico Minto,et al.  Time-of-Flight and Structured Light Depth Cameras , 2016, Springer International Publishing.

[14]  Ramesh Raskar,et al.  Fast separation of direct and global components of a scene using high frequency illumination , 2006, ACM Trans. Graph..

[15]  Olaf Hellwich,et al.  Compensation for Multipath in ToF Camera Measurements Supported by Photometric Calibration and Environment Integration , 2013, ICVS.

[16]  Alice Biber,et al.  Demodulation pixels in CCD and CMOS technologies for time-of-flight ranging , 2000, Electronic Imaging.

[17]  Gianluca Agresti,et al.  Deep Learning for Confidence Information in Stereo and ToF Data Fusion , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[18]  Min H. Kim,et al.  DeepToF: off-the-shelf real-time correction of multipath interference in time-of-flight imaging , 2017, ACM Trans. Graph..

[19]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[20]  Ming-Yu Liu,et al.  Learning to remove multipath distortions in Time-of-Flight range images for a robotic arm setup , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[21]  Reinhard Klein,et al.  Solving trigonometric moment problems for fast transient imaging , 2015, ACM Trans. Graph..

[22]  Gianluca Agresti,et al.  Combination of Spatially-Modulated ToF and Structured Light for MPI-Free Depth Estimation , 2018, ECCV Workshops.

[23]  Stefan Fuchs,et al.  Multipath Interference Compensation in Time-of-Flight Camera Images , 2010, 2010 20th International Conference on Pattern Recognition.

[24]  Roberto Manduchi,et al.  Bilateral filtering for gray and color images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[25]  Ramesh Raskar,et al.  Resolving Multi-path Interference in Time-of-Flight Imaging via Modulation Frequency Diversity and Sparse Regularization , 2014, Optics letters.

[26]  Ramesh Raskar,et al.  A light transport model for mitigating multipath interference in Time-of-flight sensors , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Ming Zhang,et al.  Multiresolution Bilateral Filtering for Image Denoising , 2008, IEEE Transactions on Image Processing.

[28]  Michael J. Cree,et al.  Review of methods for resolving multi-path interference in Time-of-Flight range cameras , 2014, IEEE SENSORS 2014 Proceedings.

[29]  Michael J. Cree,et al.  Resolving multiple propagation paths in time of flight range cameras using direct and global separation methods , 2015 .

[30]  Jan Kautz,et al.  Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset , 2018, ECCV.

[31]  Rahul Nair,et al.  Simulation of Time-of-Flight Sensors using Global Illumination , 2013, VMV.