Seeing around corners with edge-resolved transient imaging

Non-line-of-sight (NLOS) imaging is a rapidly growing field seeking to form images of objects outside the field of view, with potential applications in autonomous navigation, reconnaissance, and even medical imaging. The critical challenge of NLOS imaging is that diffuse reflections scatter light in all directions, resulting in weak signals and a loss of directional information. To address this problem, we propose a method for seeing around corners that derives angular resolution from vertical edges and longitudinal resolution from the temporal response to a pulsed light source. We introduce an acquisition strategy, scene response model, and reconstruction algorithm that enable the formation of 2.5-dimensional representations—a plan view plus heights—and a 180∘ field of view for large-scale scenes. Our experiments demonstrate accurate reconstructions of hidden rooms up to 3 meters in each dimension despite a small scan aperture (1.5-centimeter radius) and only 45 measurement locations.

[1]  Robert Henderson,et al.  Detection and tracking of moving objects hidden from view , 2015, Nature Photonics.

[2]  Xiaobin Wang,et al.  Improved algorithm of non-line-of-sight imaging based on the Bayesian statistics. , 2019, Journal of the Optical Society of America. A, Optics, image science, and vision.

[3]  R. Raskar,et al.  Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging , 2012, Nature Communications.

[4]  Leonhard Held,et al.  Gaussian Markov Random Fields: Theory and Applications , 2005 .

[5]  Jean-Yves Tourneret,et al.  Bayesian 3D Reconstruction of Complex Scenes from Single-Photon Lidar Data , 2018, SIAM J. Imaging Sci..

[6]  Ramesh Raskar,et al.  Looking around the corner using transient imaging , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[7]  Yasuhiro Mukaigawa,et al.  Polarized Non-Line-of-Sight Imaging , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  N. Reid,et al.  AN OVERVIEW OF COMPOSITE LIKELIHOOD METHODS , 2011 .

[9]  K. Eliceiri,et al.  Non-line-of-sight imaging using a time-gated single photon avalanche diode. , 2015, Optics express.

[10]  Gordon Wetzstein,et al.  Non-Line-of-Sight Surface Reconstruction Using the Directional Light-Cone Transform , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Federica Villa,et al.  Non-Line-of-Sight Three-Dimensional Imaging with a Single-Pixel Camera , 2019, Physical Review Applied.

[12]  Daniele Faccio,et al.  A Single-Shot Non-Line-of-Sight Range-Finder , 2019, Sensors.

[13]  Prasanna Rangarajan,et al.  Synthetic Wavelength Holography: An Extension of Gabor's Holographic Principle to Imaging with Scattered Wavefronts , 2019, 1912.11438.

[14]  Jean-Yves Tourneret,et al.  Real-time 3D reconstruction from single-photon lidar data using plug-and-play point cloud denoisers , 2019, Nature Communications.

[15]  Gordon Wetzstein,et al.  Acoustic Non-Line-Of-Sight Imaging , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Gordon Wetzstein,et al.  Non-line-of-sight Imaging with Partial Occluders and Surface Normals , 2017, ACM Trans. Graph..

[17]  Ramesh Raskar,et al.  Thermal Non-Line-of-Sight Imaging , 2019, 2019 IEEE International Conference on Computational Photography (ICCP).

[18]  Andreas Velten,et al.  Phasor field diffraction based reconstruction for fast non-line-of-sight imaging systems , 2020, Nature Communications.

[19]  James R. Leger,et al.  Passive non-line-of-sight imaging using plenoptic information. , 2020, Journal of the Optical Society of America. A, Optics, image science, and vision.

[20]  Gordon Wetzstein,et al.  Keyhole Imaging:Non-Line-of-Sight Imaging and Tracking of Moving Objects Along a Single Optical Path , 2019, IEEE Transactions on Computational Imaging.

[21]  Diego Gutierrez,et al.  Non-line-of-sight imaging using phasor-field virtual wave optics , 2018, Nature.

[22]  Vivek K. Goyal,et al.  Diffuse Imaging: Creating Optical Images With Unfocused Time-Resolved Illumination and Sensing , 2012, IEEE Signal Processing Letters.

[23]  Vivek K. Goyal,et al.  Multi-Depth Computational Periscopy with an Ordinary Camera , 2020, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[24]  P. Green Reversible jump Markov chain Monte Carlo computation and Bayesian model determination , 1995 .

[25]  T FreemanWilliam,et al.  Accidental Pinhole and Pinspeck Cameras , 2014 .

[26]  Gordon Wetzstein,et al.  Confocal non-line-of-sight imaging based on the light-cone transform , 2018, Nature.

[27]  J. G. Skellam The frequency distribution of the difference between two Poisson variates belonging to different populations. , 1946, Journal of the Royal Statistical Society. Series A.

[28]  Matthias B. Hullin,et al.  Non-line-of-sight Reconstruction Using Efficient Transient Rendering , 2018, ACM Trans. Graph..

[29]  Christos Thrampoulidis,et al.  Using Unknown Occluders to Recover Hidden Scenes , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  A. Baddeley,et al.  Area-interaction point processes , 1993 .

[31]  Antonio Torralba,et al.  Accidental pinhole and pinspeck cameras: Revealing the scene outside the picture , 2012, CVPR.

[32]  Felix Heide,et al.  Seeing Around Street Corners: Non-Line-of-Sight Detection and Tracking In-the-Wild Using Doppler Radar , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Jeffrey H. Shapiro,et al.  Exploiting Occlusion in Non-Line-of-Sight Active Imaging , 2017, IEEE Transactions on Computational Imaging.

[34]  Christos Thrampoulidis,et al.  Revealing hidden scenes by photon-efficient occlusion-based opportunistic active imaging. , 2018, Optics express.

[35]  Aswin C. Sankaranarayanan,et al.  Beyond Volumetric Albedo — A Surface Optimization Framework for Non-Line-Of-Sight Imaging , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Kiriakos N. Kutulakos,et al.  A Theory of Fermat Paths for Non-Line-Of-Sight Shape Reconstruction , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Vivek K. Goyal,et al.  Corner Occluder Computational Periscopy: Estimating a Hidden Scene from a Single Photograph , 2019, 2019 IEEE International Conference on Computational Photography (ICCP).

[38]  William T. Freeman,et al.  Two-Dimensional Non-Line-of-Sight Scene Estimation From a Single Edge Occluder , 2020, IEEE Transactions on Computational Imaging.

[39]  Diego Gutierrez,et al.  Fast back-projection for non-line of sight reconstruction , 2017, SIGGRAPH Posters.

[40]  Andrew Gelman,et al.  Handbook of Markov Chain Monte Carlo , 2011 .

[41]  Ashok Veeraraghavan,et al.  Reconstructing rooms using photon echoes: A plane based model and reconstruction algorithm for looking around the corner , 2017, 2017 IEEE International Conference on Computational Photography (ICCP).

[42]  Felix Heide,et al.  Steady-State Non-Line-Of-Sight Imaging , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Aswin C. Sankaranarayanan,et al.  Convolutional Approximations to the General Non-Line-of-Sight Imaging Operator , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[44]  Wolfgang Heidrich,et al.  Diffuse Mirrors: 3D Reconstruction from Diffuse Indirect Illumination Using Inexpensive Time-of-Flight Sensors , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[45]  Vivek K Goyal,et al.  Advances in Single-Photon Lidar for Autonomous Vehicles: Working Principles, Challenges, and Recent Advances , 2020, IEEE Signal Processing Magazine.

[46]  Antonio Torralba,et al.  Accidental Pinhole and Pinspeck Cameras , 2014, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[47]  Frédo Durand,et al.  Turning Corners into Cameras: Principles and Methods , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[48]  Gordon Wetzstein,et al.  Wave-based non-line-of-sight imaging using fast f-k migration , 2019, ACM Trans. Graph..

[49]  Felix Heide,et al.  Deep-inverse correlography: towards real-time high-resolution non-line-of-sight imaging , 2020, Optica.

[50]  Charles Saunders,et al.  Computational periscopy with an ordinary digital camera , 2019, Nature.

[51]  Xiaobai Sun,et al.  Reference structure tomography. , 2004, Journal of the Optical Society of America. A, Optics, image science, and vision.

[52]  Frédo Durand,et al.  Inferring Light Fields from Shadows , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.