Lightweight Deep Learning Architecture for MPI Correction and Transient Reconstruction

Indirect Time-of-Flight cameras (iToF) are low-cost devices that provide depth images at an interactive frame rate. However, they are affected by different error sources, with the spotlight taken by Multi-Path Interference (MPI), a key challenge for this technology. Common data-driven approaches tend to focus on a direct estimation of the output depth values, ignoring the underlying transient propagation of the light in the scene. In this work instead, we propose a very compact architecture, leveraging on the direct-global subdivision of transient information for the removal of MPI and for the reconstruction of the transient information itself. The proposed model reaches state-of-the-art MPI correction performances both on synthetic and real data and proves to be very competitive also at extreme levels of noise; at the same time, it also makes a step towards reconstructing transient information from multi-frequency iToF data.

[1]  Jake K. Aggarwal,et al.  Structure from stereo-a review , 1989, IEEE Trans. Syst. Man Cybern..

[2]  Erkan Bostanci,et al.  Augmented reality applications for cultural heritage using Kinect , 2015, Human-centric Computing and Information Sciences.

[3]  Farzin Amzajerdian,et al.  Lidar systems for precision navigation and safe landing on planetary bodies , 2011, Other Conferences.

[4]  Qionghai Dai,et al.  Fourier Analysis on Transient Imaging with a Multifrequency Time-of-Flight Camera , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Zhiwei Xiong,et al.  Spatial Hierarchy Aware Residual Pyramid Network for Time-of-Flight Depth Denoising , 2020, ECCV.

[6]  Stefan Fuchs,et al.  Multipath Interference Compensation in Time-of-Flight Camera Images , 2010, 2010 20th International Conference on Pattern Recognition.

[7]  Gianluca Agresti,et al.  Deep Learning for Transient Image Reconstruction from ToF Data , 2021, Sensors.

[8]  Young Min Kim,et al.  Multi-view image and ToF sensor fusion for dense 3D reconstruction , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[9]  Piergiorgio Sartor,et al.  Unsupervised Domain Adaptation for ToF Data Denoising With Adversarial Learning , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Diego Gutierrez,et al.  A data-driven compression method for transient rendering , 2019, SIGGRAPH Posters.

[11]  Yan Wang,et al.  Pseudo-LiDAR From Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Qingquan Li,et al.  3D LIDAR point cloud based intersection recognition for autonomous driving , 2012, 2012 IEEE Intelligent Vehicles Symposium.

[13]  Gordon Wetzstein,et al.  A dataset for benchmarking time-resolved non-line-of-sight imaging , 2019, SIGGRAPH Posters.

[14]  L. Guibas,et al.  The Earth Mover''s Distance: Lower Bounds and Invariance under Translation , 1997 .

[15]  Gianluca Agresti,et al.  Deep Learning for Multi-path Error Removal in ToF Sensors , 2018, ECCV Workshops.

[16]  Wolfgang Heidrich,et al.  Diffuse Mirrors: 3D Reconstruction from Diffuse Indirect Illumination Using Inexpensive Time-of-Flight Sensors , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Andreas Velten,et al.  iToF2dToF: A Robust and Flexible Representation for Data-Driven Time-of-Flight Imaging , 2021, IEEE Transactions on Computational Imaging.

[18]  Min H. Kim,et al.  DeepToF: off-the-shelf real-time correction of multipath interference in time-of-flight imaging , 2017, ACM Trans. Graph..

[19]  Olaf Hellwich,et al.  Compensation for Multipath in ToF Camera Measurements Supported by Photometric Calibration and Environment Integration , 2013, ICVS.

[20]  Manuel Mazo,et al.  Modeling and correction of multipath interference in time of flight cameras , 2014, Image Vis. Comput..

[21]  W. Weibull A Statistical Distribution Function of Wide Applicability , 1951 .

[22]  Greg Humphreys,et al.  Physically Based Rendering: From Theory to Implementation , 2004 .

[23]  Ramesh Raskar,et al.  Resolving Multipath Interference in Kinect: An Inverse Problem Approach , 2014, IEEE Sensors Journal.

[24]  Jan Kautz,et al.  Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset , 2018, ECCV.

[25]  Mirko Schmidt,et al.  SRA: Fast Removal of General Multipath for ToF Sensors , 2014, ECCV.

[26]  Gordon Wetzstein,et al.  Deep End-to-End Time-of-Flight Imaging , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[27]  R. Dubayah,et al.  Lidar Remote Sensing for Forestry , 2000, Journal of Forestry.

[28]  HoraudRadu,et al.  An overview of depth cameras and range scanners based on time-of-flight technologies , 2016 .

[29]  Ludovico Minto,et al.  Time-of-Flight and Structured Light Depth Cameras , 2016, Springer International Publishing.