A Theory of Fermat Paths for Non-Line-Of-Sight Shape Reconstruction

We present a novel theory of Fermat paths of light between a known visible scene and an unknown object not in the line of sight of a transient camera. These light paths either obey specular reflection or are reflected by the object's boundary, and hence encode the shape of the hidden object. We prove that Fermat paths correspond to discontinuities in the transient measurements. We then derive a novel constraint that relates the spatial derivatives of the path lengths at these discontinuities to the surface normal. Based on this theory, we present an algorithm, called Fermat Flow, to estimate the shape of the non-line-of-sight object. Our method allows, for the first time, accurate shape recovery of complex objects, ranging from diffuse to specular, that are hidden around the corner as well as hidden behind a diffuser. Finally, our approach is agnostic to the particular technology used for transient imaging. As such, we demonstrate mm-scale shape recovery from pico-second scale transients using a SPAD and ultrafast laser, as well as micron-scale reconstruction from femto-second scale transients using interferometry. We believe our work is a significant advance over the state-of-the-art in non-line-of-sight imaging.

[1]  Matthew O'Toole,et al.  Tracking Multiple Objects Outside the Line of Sight Using Speckle Imaging , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[2]  R. Rezvani,et al.  Passive sensing around the corner using spatial coherence , 2018, Nature Communications.

[3]  Min Chen,et al.  Theory and application of specular path perturbation , 2000, TOGS.

[4]  Gordon Wetzstein,et al.  Acoustic Non-Line-Of-Sight Imaging , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Min Chen,et al.  Perturbation Methods for Interactive Specular Reflections , 2000, IEEE Trans. Vis. Comput. Graph..

[6]  Philippe Bekaert,et al.  Advanced global illumination , 2006 .

[7]  Ramesh Raskar,et al.  Flash Photography for Data-Driven Hidden Scene Recovery , 2018, ArXiv.

[8]  Daniel Mayost.  Applications Of The Signed Distance Function To Surface Geometry , 2014 .

[9]  J. A. Sethian,et al.  Fast Marching Methods , 1999, SIAM Rev..

[10]  Yuichi Taguchi,et al.  Beyond Alhazen's problem: Analytical projection model for non-central catadioptric cameras with quadric mirrors , 2011, CVPR 2011.

[11]  Andreas Velten,et al.  Analysis of Feature Visibility in Non-Line-Of-Sight Measurements , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Wolfgang Heidrich,et al.  Diffuse Mirrors: 3D Reconstruction from Diffuse Indirect Illumination Using Inexpensive Time-of-Flight Sensors , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[13]  Ramesh Raskar,et al.  Occluded Imaging with Time-of-Flight Sensors , 2016, ACM Trans. Graph..

[14]  J. Bertolotti,et al.  Non-invasive imaging through opaque scattering layers , 2012, Nature.

[15]  Ashok Veeraraghavan,et al.  Reconstructing rooms using photon echoes: A plane based model and reconstruction algorithm for looking around the corner , 2017, 2017 IEEE International Conference on Computational Photography (ICCP).

[16]  J. Sethian,et al.  3-D traveltime computation using the fast marching method , 1999 .

[17]  Michael M. Kazhdan,et al.  Poisson surface reconstruction , 2006, SGP '06.

[18]  O. Katz,et al.  Looking around corners and through thin turbid layers in real time with scattered incoherent light , 2012, Nature Photonics.

[19]  Jeffrey H. Shapiro,et al.  Exploiting Occlusion in Non-Line-of-Sight Active Imaging , 2017, IEEE Transactions on Computational Imaging.

[20]  Diego Gutierrez,et al.  Recent advances in transient imaging: A computer graphics and vision perspective , 2016, Vis. Informatics.

[21]  Gordon Wetzstein,et al.  Robust Non-line-of-sight Imaging with Single Photon Detectors , 2017, ArXiv.

[22]  Shuang Zhao,et al.  Single scattering in refractive media with triangle mesh boundaries , 2009, SIGGRAPH '09.

[23]  R. Raskar,et al.  Single-photon sensitive light-in-fight imaging , 2015, Nature Communications.

[24]  Aswin C. Sankaranarayanan,et al.  Beyond Volumetric Albedo — A Surface Optimization Framework for Non-Line-Of-Sight Imaging , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Orestes N. Stavroudis.  The Optics of Rays, Wavefronts, and Caustics , 2012 .

[26]  R. Raskar,et al.  Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging , 2012, Nature Communications.

[27]  K. Eliceiri,et al.  Non-line-of-sight imaging using a time-gated single photon avalanche diode. , 2015, Optics express.

[28]  P. Giblin,et al.  Curves and singularities : a geometrical introduction to singularity theory , 1992 .

[29]  Diego Gutierrez,et al.  A Computational Model of a Single-Photon Avalanche Diode Sensor for Transient Imaging , 2017, ArXiv.

[30]  Yuichi Taguchi,et al.  Analytical Forward Projection for Axial Non-central Dioptric and Catadioptric Cameras , 2010, ECCV.

[31]  Frédo Durand,et al.  Inferring Light Fields from Shadows , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[32]  Shree K. Nayar,et al.  A Theory of Single-Viewpoint Catadioptric Image Formation , 1999, International Journal of Computer Vision.

[33]  Christos Thrampoulidis,et al.  Revealing hidden scenes by photon-efficient occlusion-based opportunistic active imaging. , 2018, Optics express.

[34]  Diego Gutierrez,et al.  Fast back-projection for non-line of sight reconstruction , 2017, SIGGRAPH Posters.

[35]  Ramesh Raskar,et al.  Reconstruction of hidden 3D shapes using diffuse reflections , 2012, Optics express.

[36]  Frédo Durand,et al.  Micron-scale light transport decomposition using interferometry , 2015, ACM Trans. Graph..

[37]  M. Spivak.  Calculus On Manifolds: A Modern Approach To Classical Theorems Of Advanced Calculus , 2019 .

[38]  James A. Sethian,et al.  Level Set Methods and Fast Marching Methods: Evolving Interfaces in Computational Geometry, Fluid , 2012 .

[39]  Charles Saunders,et al.  Computational periscopy with an ordinary digital camera , 2019, Nature.

[40]  Diego Gutierrez,et al.  A framework for transient rendering , 2014, ACM Trans. Graph..

[41]  Kiriakos N. Kutulakos,et al.  The Geometry of First-Returning Photons for Non-Line-of-Sight Imaging , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Diego Gutierrez,et al.  Femto-photography , 2013, ACM Trans. Graph..

[43]  Qionghai Dai,et al.  Ultra-fast Lensless Computational Imaging through 5D Frequency Analysis of Time-resolved Light Transport , 2013, International Journal of Computer Vision.

[44]  Pat Hanrahan,et al.  Illumination from curved reflectors , 1992, SIGGRAPH.

[45]  Gordon Wetzstein,et al.  Confocal non-line-of-sight imaging based on the light-cone transform , 2018, Nature.

[46]  J. W. Bruce,et al.  On caustics of plane curves , 1981 .

[47]  Michael M. Kazhdan,et al.  Screened poisson surface reconstruction , 2013, TOGS.

[48]  Gordon Wetzstein,et al.  Reconstructing Transient Images from Single-Photon Sensors , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Ashok Veeraraghavan,et al.  SNLOS: Non-line-of-sight Scanning through Temporal Focusing , 2019, 2019 IEEE International Conference on Computational Photography (ICCP).

[50]  Ronald Fedkiw,et al.  Level set methods and dynamic implicit surfaces , 2002, Applied mathematical sciences.

[51]  Steve Marschner,et al.  Manifold exploration , 2012, ACM Trans. Graph..

[52]  Frédo Durand,et al.  Turning Corners into Cameras: Principles and Methods , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[53]  M. Fink,et al.  Non-invasive single-shot imaging through scattering layers and around corners via speckle correlations , 2014, Nature Photonics.

[54]  Ori Katz,et al.  Passive optical time-of-flight for non line-of-sight localization , 2018, Nature Communications.

[55]  Jaime Martín,et al.  Tracking objects outside the line of sight using 2D intensity images , 2016, Scientific Reports.

[56]  Ramesh Raskar,et al.  Looking Around the Corner using Ultrafast Transient Imaging , 2011, International Journal of Computer Vision.

[57]  Andrew Blake,et al.  The information available to a moving observer from specularities , 1989, Image Vis. Comput..

[58]  Matthew O'Toole,et al.  Primal-dual coding to probe light transport , 2012, ACM Trans. Graph..