Combination of Spatially-Modulated ToF and Structured Light for MPI-Free Depth Estimation

Multi-path Interference (MPI) is one of the major sources of error in Time-of-Flight (ToF) camera depth measurements. A possible solution for its removal is based on the separation of direct and global light through the projection of multiple sinusoidal patterns. In this work we extend this approach by applying a Structured Light (SL) technique on the same projected patterns. This allows to compute two depth maps with a single ToF acquisition, one with the Time-of-Flight principle and the other with the Structured Light principle. The two depth fields are finally combined using a Maximum-Likelihood approach in order to obtain an accurate depth estimation free from MPI error artifacts. Experimental results demonstrate that the proposed method has very good MPI correction properties with state-of-the-art performances.

[1]  Pietro Zanuttigh,et al.  Head-mounted gesture controlled interface for human-computer interaction , 2016, Multimedia Tools and Applications.

[2]  Michael J. Cree,et al.  Resolving multiple propagation paths in time of flight range cameras using direct and global separation methods , 2015 .

[3]  Olaf Hellwich,et al.  Compensation for Multipath in ToF Camera Measurements Supported by Photometric Calibration and Environment Integration , 2013, ICVS.

[4]  S. Hussmann,et al.  Robot Vision System based on a 3D-TOF Camera , 2007, 2007 IEEE Instrumentation & Measurement Technology Conference IMTC 2007.

[5]  Ramesh Raskar,et al.  A light transport model for mitigating multipath interference in Time-of-flight sensors , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Robert Mahony,et al.  Statistical analysis of measurement processes for time-of-flight cameras , 2009, Optical Engineering + Applications.

[7]  L. Van Gool,et al.  Combining RGB and ToF cameras for real-time 3D hand gesture interaction , 2011, 2011 IEEE Workshop on Applications of Computer Vision (WACV).

[8]  Song Zhang,et al.  Phase error compensation for three-dimensional shape measurement with projector defocusing. , 2011, Applied optics.

[9]  Guido M. Cortelazzo,et al.  A Probabilistic Approach to ToF and Stereo Data Fusion , 2010 .

[10]  Stefan Fuchs,et al.  Multipath Interference Compensation in Time-of-Flight Camera Images , 2010, 2010 20th International Conference on Pattern Recognition.

[11]  Guido M. Cortelazzo,et al.  Probabilistic ToF and Stereo Data Fusion Based on Mixed Pixels Measurement Models , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  艾德里安·安德鲁·多林顿,et al.  Time of flight camera system which resolves direct and multi-path radiation components , 2015 .

[13]  Ruigang Yang,et al.  Spatial-Temporal Fusion for High Accuracy Depth Maps Using Dynamic MRFs , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Alice Biber,et al.  Demodulation pixels in CCD and CMOS technologies for time-of-flight ranging , 2000, Electronic Imaging.

[15]  Michael J. Cree,et al.  Review of methods for resolving multi-path interference in Time-of-Flight range cameras , 2014, IEEE SENSORS 2014 Proceedings.

[16]  Nassir Navab,et al.  Human skeleton tracking from depth data using geodesic distances and optical flow , 2012, Image Vis. Comput..

[17]  Ming-Yu Liu,et al.  Learning to remove multipath distortions in Time-of-Flight range images for a robotic arm setup , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[18]  Gianluca Agresti,et al.  Deep Learning for Confidence Information in Stereo and ToF Data Fusion , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[19]  Ramesh Raskar,et al.  Fast separation of direct and global components of a scene using high frequency illumination , 2006, SIGGRAPH 2006.

[20]  Ramesh Raskar,et al.  Resolving Multi-path Interference in Time-of-Flight Imaging via Modulation Frequency Diversity and Sparse Regularization , 2014, Optics letters.

[21]  Min H. Kim,et al.  DeepToF: off-the-shelf real-time correction of multipath interference in time-of-flight imaging , 2017, ACM Trans. Graph..

[22]  Gordon Wetzstein,et al.  Deep End-to-End Time-of-Flight Imaging , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[23]  Mirko Schmidt,et al.  SRA: Fast Removal of General Multipath for ToF Sensors , 2014, ECCV.

[24]  Ramesh Raskar,et al.  Coded time of flight cameras , 2013, ACM Trans. Graph..

[25]  Gianluca Agresti,et al.  Deep Learning for Multi-path Error Removal in ToF Sensors , 2018, ECCV Workshops.

[26]  Shree K. Nayar,et al.  Micro Phase Shifting , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[27]  Jan Kautz,et al.  Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset , 2018, ECCV.

[28]  Rahul Nair,et al.  Simulation of Time-of-Flight Sensors using Global Illumination , 2013, VMV.

[29]  Peter Seitz,et al.  The lock-in CCD-two-dimensional synchronous detection of light , 1995 .

[30]  Matthew O'Toole,et al.  Temporal frequency probing for 5D transient analysis of global light transport , 2014, ACM Trans. Graph..

[31]  Mirko Schmidt.  Analysis, Modeling and Dynamic Optimization of 3D Time-of-Flight Imaging Systems , 2011 .

[32]  Manuel Mazo,et al.  Modeling and correction of multipath interference in time of flight cameras , 2014, Image Vis. Comput..

[33]  MOHIT GUPTA,et al.  Phasor Imaging , 2015, ACM Trans. Graph..

[34]  Ludovico Minto,et al.  Time-of-Flight and Structured Light Depth Cameras , 2016, Springer International Publishing.