A light transport model for mitigating multipath interference in Time-of-flight sensors

Continuous-wave Time-of-flight (TOF) range imaging has become a commercially viable technology with many applications in computer vision and graphics. However, the depth images obtained from TOF cameras contain scene dependent errors due to multipath interference (MPI). Specifically, MPI occurs when multiple optical reflections return to a single spatial location on the imaging sensor. Many prior approaches to rectifying MPI rely on sparsity in optical reflections, which is an extreme simplification. In this paper, we correct MPI by combining the standard measurements from a TOF camera with information from direct and global light transport. We report results on both simulated experiments and physical experiments (using the Kinect sensor). Our results, evaluated against ground truth, demonstrate a quantitative improvement in depth accuracy.

[1]  Hans-Peter Seidel,et al.  Dynamic Display of BRDFs , 2011, Comput. Graph. Forum.

[2]  Ramesh Raskar,et al.  Estimating wide-angle, spatially varying reflectance using time-resolved inversion of backscattered light. , 2014, Journal of the Optical Society of America. A, Optics, image science, and vision.

[3]  Ramesh Raskar,et al.  Single view reflectance capture using multiplexed scattering and time-of-flight imaging , 2011, SA '11.

[4]  Ashok Veeraraghavan,et al.  A Practical Approach to 3D Scanning in the Presence of Interreflections, Subsurface Scattering and Defocus , 2013, International Journal of Computer Vision.

[5]  Ramesh Raskar,et al.  Coded Time-of-Flight Imaging for Calibration Free Fluorescence Lifetime Estimation , 2014 .

[6]  Gordon Wetzstein,et al.  The visual computing of projector-camera systems , 2008, SIGGRAPH '08.

[7]  Michael J. Cree,et al.  Closed-form inverses for the mixed pixel/multipath interference problem in AMCW lidar , 2012, Electronic Imaging.

[8]  Mohit Gupta,et al.  Legendre fluids: a unified framework for analytic reduced space modeling and rendering of participating media , 2007, SCA '07.

[9]  Ramesh Raskar,et al.  3D Depth Cameras in Vision: Benefits and Limitations of the Hardware , 2014 .

[10]  Hans-Peter Seidel,et al.  Acquisition and analysis of bispectral bidirectional reflectance and reradiation distribution functions , 2010, SIGGRAPH 2010.

[11]  H. Seidel,et al.  Fluorescent immersion range scanning , 2008, ACM Trans. Graph..

[12]  Mirko Schmidt,et al.  SRA: Fast Removal of General Multipath for ToF Sensors , 2014, ECCV.

[13]  Stefan Fuchs,et al.  Multipath Interference Compensation in Time-of-Flight Camera Images , 2010, 2010 20th International Conference on Pattern Recognition.

[14]  Frédo Durand,et al.  Noise-optimal capture for high dynamic range photography , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[15]  Guido M. Cortelazzo,et al.  Time-of-Flight Cameras and Microsoft Kinect™ , 2012, Springer Briefs in Electrical and Computer Engineering.

[16]  Matthew O'Toole,et al.  3D Shape and Indirect Appearance by Structured Light Transport , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Philip A. Chou,et al.  SPUMIC: Simultaneous phase unwrapping and multipath interference cancellation in time-of-flight cameras using spectral methods , 2013, 2013 IEEE International Conference on Multimedia and Expo (ICME).

[18]  Wolfgang Heidrich,et al.  Diffuse Mirrors: 3D Reconstruction from Diffuse Indirect Illumination Using Inexpensive Time-of-Flight Sensors , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Ramesh Raskar,et al.  Fast separation of direct and global components of a scene using high frequency illumination , 2006, ACM Trans. Graph..

[20]  Shree K. Nayar,et al.  Micro Phase Shifting , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Luc Van Gool,et al.  Face/Off: live facial puppetry , 2009, SCA '09.

[22]  Manuel Mazo,et al.  Modeling and correction of multipath interference in time of flight cameras , 2014, Image Vis. Comput..

[23]  MOHIT GUPTA,et al.  Phasor Imaging , 2015, ACM Trans. Graph..

[24]  Shree K. Nayar,et al.  Diffuse structured light , 2012, 2012 IEEE International Conference on Computational Photography (ICCP).

[25]  John Sell,et al.  The Xbox One System on a Chip and Kinect Sensor , 2014, IEEE Micro.

[26]  Qionghai Dai,et al.  Frequency Analysis of Transient Light Transport with Applications in Bare Sensor Imaging , 2012, ECCV.

[27]  Tian-Tsong Ng,et al.  On the Duality of Forward and Inverse Light Transport , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Wolfgang Heidrich,et al.  Low-budget transient imaging using photonic mixer devices , 2013, ACM Trans. Graph..

[29]  Matthew O'Toole,et al.  Primal-dual coding to probe light transport , 2012, ACM Trans. Graph..

[30]  Shree K. Nayar,et al.  Structured Light in Sunlight , 2013, 2013 IEEE International Conference on Computer Vision.

[31]  VeeraraghavanAshok,et al.  A Practical Approach to 3D Scanning in the Presence of Interreflections, Subsurface Scattering and Defocus , 2013 .

[32]  Kiriakos N. Kutulakos,et al.  A theory of inverse light transport , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[33]  Ramesh Raskar,et al.  Coded time of flight cameras , 2013, ACM Trans. Graph..

[34]  Radu Horaud,et al.  Time-of-Flight Cameras , 2012, SpringerBriefs in Computer Science.

[35]  Matthew O'Toole,et al.  Temporal frequency probing for 5D transient analysis of global light transport , 2014, ACM Trans. Graph..

[36]  Shree K. Nayar,et al.  Multiplexed illumination for scene recovery in the presence of global illumination , 2011, 2011 International Conference on Computer Vision.

[37]  Srinivasa G. Narasimhan,et al.  Compensating for Motion during Direct-Global Separation , 2013, 2013 IEEE International Conference on Computer Vision.

[38]  Ramesh Raskar,et al.  Demultiplexing illumination via low cost sensing and nanosecond coding , 2014, 2014 IEEE International Conference on Computational Photography (ICCP).

[39]  Dikpal Reddy,et al.  Frequency-Space Decomposition and Acquisition of Light Transport under Spatially Varying Illumination , 2012, ECCV.

[40]  Tim Weyrich,et al.  Computational Fabrication and Display of Material Appearance , 2013, Eurographics.

[41]  Ramesh Raskar,et al.  Resolving Multi-path Interference in Time-of-Flight Imaging via Modulation Frequency Diversity and Sparse Regularization , 2014, Optics letters.

[42]  Mohit Gupta,et al.  Shape from Scatter , 2014, Computer Vision, A Reference Guide.

[43]  S. Foix,et al.  Lock-in Time-of-Flight (ToF) Cameras: A Survey , 2011, IEEE Sensors Journal.

[44]  Michael J. Cree,et al.  Separating true range measurements from multi-path and scattering interference in commercial range cameras , 2011, Electronic Imaging.

[45]  Mohit Gupta,et al.  On controlling light transport in poor visibility environments , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[46]  James Davis,et al.  5d time-light transport matrix: What can we reason about scene properties? , 2008 .

[47]  A. J. Schwemin,et al.  Pose Estimation Using Time-resolved Inversion of Diffuse Light References and Links , 2022 .

[48]  Shree K. Nayar,et al.  Structured light in scattering media , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[49]  Tony Q. S. Quek,et al.  Radiometric compensation using stratified inverses , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[50]  Ramesh Raskar,et al.  Multifrequency time of flight in the context of transient renderings , 2013, SIGGRAPH '13.

[51]  Steve Marschner,et al.  Rendering glints on high-resolution normal-mapped specular surfaces , 2014, ACM Trans. Graph..