Learned feature embeddings for non-line-of-sight imaging and recognition

Objects obscured by occluders are considered lost in the images acquired by conventional camera systems, prohibiting both visualization and understanding of such hidden objects. Non-line-of-sight methods (NLOS) aim at recovering information about hidden scenes, which could help make medical imaging less invasive, improve the safety of autonomous vehicles, and potentially enable capturing unprecedented high-definition RGB-D data sets that include geometry beyond the directly visible parts. Recent NLOS methods have demonstrated scene recovery from time-resolved pulse-illuminated measurements encoding occluded objects as faint indirect reflections. Unfortunately, these systems are fundamentally limited by the quartic intensity fall-off for diffuse scenes. With laser illumination limited by eye-safety limits, recovery algorithms must tackle this challenge by incorporating scene priors. However, existing NLOS reconstruction algorithms do not facilitate learning scene priors. Even if they did, datasets that allow for such supervision do not exist, and successful encoder-decoder networks and generative adversarial networks fail for real-world NLOS data. In this work, we close this gap by learning hidden scene feature representations tailored to both reconstruction and recognition tasks such as classification or object detection, while still relying on physical models at the feature level. We overcome the lack of real training data with a generalizable architecture that can be trained in simulation. We learn the differentiable scene representation jointly with the reconstruction task using a differentiable transient renderer in the objective, and demonstrate that it generalizes to unseen classes and unseen real-world scenes, unlike existing encoder-decoder architectures and generative adversarial networks. The proposed method allows for end-to-end training for different NLOS tasks, such as image reconstruction, classification, and object detection, while being memory-efficient and running at real-time rates. We demonstrate hidden view synthesis, RGB-D reconstruction, classification, and object detection in the hidden scene in an end-to-end fashion.

[1]  P. B. Coates Pile-up corrections in the measurement of lifetimes , 1972 .

[2]  R. Stolt MIGRATION BY FOURIER TRANSFORM , 1978 .

[3]  N. Abramson Light-in-flight recording by holography. , 1978, Optics letters.

[4]  Donald P. Greenberg,et al.  The hemi-cube: a radiosity solution for complex environments , 1985, SIGGRAPH.

[5]  Ramesh Raskar,et al.  Looking around the corner using transient imaging , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[6]  Ramesh Raskar,et al.  Estimating Motion and size of moving non-line-of-sight objects in cluttered environments , 2011, CVPR 2011.

[7]  K. Bala,et al.  Single view reflectance capture using multiplexed scattering and time-of-flight imaging , 2011, ACM Trans. Graph..

[8]  Ramesh Raskar,et al.  Single view reflectance capture using multiplexed scattering and time-of-flight imaging , 2011, SA '11.

[9]  R. Raskar,et al.  Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging , 2012, Nature Communications.

[10]  O. Katz,et al.  Looking around corners and through thin turbid layers in real time with scattered incoherent light , 2012, Nature Photonics.

[11]  Qionghai Dai,et al.  Decomposing Global Light Transport Using Time of Flight Imaging , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Ramesh Raskar,et al.  Reconstruction of hidden 3D shapes using diffuse reflections , 2012, Optics express.

[13]  Wolfgang Heidrich,et al.  Low-budget transient imaging using photonic mixer devices , 2013, ACM Trans. Graph..

[14]  Ramesh Raskar,et al.  Coded time of flight cameras , 2013, ACM Trans. Graph..

[15]  Diego Gutierrez,et al.  Femto-photography , 2013, ACM Trans. Graph..

[16]  Diego Gutierrez,et al.  A framework for transient rendering , 2014, ACM Trans. Graph..

[17]  R. Howe,et al.  17th International Conference on Medical Image Computing and Computer-Assisted Intervention. , 2014, Medical image computing and computer-assisted intervention : MICCAI ... International Conference on Medical Image Computing and Computer-Assisted Intervention.

[18]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[19]  Andreas Velten,et al.  Feature selection and back-projection algorithms for nonline-of-sight laser–gated viewing , 2014, J. Electronic Imaging.

[20]  Andrew J. Holmes,et al.  A 9.8 μm sample and hold time to amplitude converter CMOS SPAD pixel , 2014, 2014 44th European Solid State Device Research Conference (ESSDERC).

[21]  Vivek K Goyal,et al.  First-Photon Imaging , 2014, Science.

[22]  M. Fink,et al.  Non-invasive single-shot imaging through scattering layers and around corners via speckle correlations , 2014, Nature Photonics.

[23]  Wolfgang Heidrich,et al.  Diffuse Mirrors: 3D Reconstruction from Diffuse Indirect Illumination Using Inexpensive Time-of-Flight Sensors , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  K. Eliceiri,et al.  Non-line-of-sight imaging using a time-gated single photon avalanche diode. , 2015, Optics express.

[25]  Thomas Brox,et al.  Single-view to Multi-view: Reconstructing Unseen Views with a Convolutional Network , 2015, ArXiv.

[26]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[27]  Jianxiong Xiao,et al.  3D ShapeNets: A deep representation for volumetric shapes , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Leonidas J. Guibas,et al.  ShapeNet: An Information-Rich 3D Model Repository , 2015, ArXiv.

[29]  Andrew Zisserman,et al.  Spatial Transformer Networks , 2015, NIPS.

[30]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Jitendra Malik,et al.  View Synthesis by Appearance Flow , 2016, ECCV.

[32]  Samuel Burri,et al.  Challenges and Solutions to Next-Generation Single-Photon Imagers , 2016 .

[33]  Thomas Brox,et al.  3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation , 2016, MICCAI.

[34]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[35]  Silvio Savarese,et al.  3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction , 2016, ECCV.

[36]  Ramesh Raskar,et al.  Occluded Imaging with Time-of-Flight Sensors , 2016, ACM Trans. Graph..

[37]  Jaime Martín,et al.  Tracking objects outside the line of sight using 2D intensity images , 2016, Scientific Reports.

[38]  Min H. Kim,et al.  DeepToF: off-the-shelf real-time correction of multipath interference in time-of-flight imaging , 2017, ACM Trans. Graph..

[39]  Kiriakos N. Kutulakos,et al.  The Geometry of First-Returning Photons for Non-Line-of-Sight Imaging , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Jonathan Leach,et al.  Non-line-of-sight tracking of people at long range , 2017, Optics express.

[41]  Diego Gutierrez,et al.  Fast back-projection for non-line of sight reconstruction , 2017, SIGGRAPH Posters.

[42]  Diego Gutierrez,et al.  Recent advances in transient imaging: A computer graphics and vision perspective , 2016, Vis. Informatics.

[43]  Frédo Durand,et al.  Turning Corners into Cameras: Principles and Methods , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[44]  DeepToF , 2017 .

[45]  Ashok Veeraraghavan,et al.  Reconstructing rooms using photon echoes: A plane based model and reconstruction algorithm for looking around the corner , 2017, 2017 IEEE International Conference on Computational Photography (ICCP).

[46]  Diego Gutierrez,et al.  A Computational Model of a Single-Photon Avalanche Diode Sensor for Transient Imaging , 2017, ArXiv.

[47]  Gordon Wetzstein,et al.  Confocal non-line-of-sight imaging based on the light-cone transform , 2018, Nature.

[48]  Stefan Roth,et al.  Matryoshka Networks: Predicting 3D Geometry via Nested Shape Layers , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[49]  Jaakko Lehtinen,et al.  Progressive Growing of GANs for Improved Quality, Stability, and Variation , 2017, ICLR.

[50]  Adrián Jarabo,et al.  Bidirectional Rendering of Vector Light Transport , 2018, Comput. Graph. Forum.

[51]  Gordon Wetzstein,et al.  Deep End-to-End Time-of-Flight Imaging , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[52]  Ramesh Raskar,et al.  Flash Photography for Data-Driven Hidden Scene Recovery , 2018, ArXiv.

[53]  Jan Kautz,et al.  Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset , 2018, ECCV.

[54]  Daniel Buschek,et al.  Neural network identification of people hidden from view with a single-pixel, single-photon detector , 2017, Scientific Reports.

[55]  F. Nolet,et al.  Quenching Circuit and SPAD Integrated in CMOS 65 nm with 7.8 ps FWHM Single Photon Timing Resolution , 2018, Instruments.

[56]  Gordon Wetzstein,et al.  Sub-picosecond photon-efficient 3D imaging using single-photon sensors , 2018, Scientific Reports.

[57]  Christos Thrampoulidis,et al.  Revealing hidden scenes by photon-efficient occlusion-based opportunistic active imaging. , 2018, Optics express.

[58]  Sergey Tulyakov,et al.  Transformable Bottleneck Networks , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[59]  Charles Saunders,et al.  Computational periscopy with an ordinary digital camera , 2019, Nature.

[60]  Aswin C. Sankaranarayanan,et al.  Beyond Volumetric Albedo — A Surface Optimization Framework for Non-Line-Of-Sight Imaging , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[61]  Gordon Wetzstein,et al.  Wave-based non-line-of-sight imaging using fast f-k migration , 2019, ACM Trans. Graph..

[62]  Diego Gutierrez,et al.  Non-line-of-sight imaging using phasor-field virtual wave optics , 2018, Nature.

[63]  Yaser Sheikh,et al.  Neural volumes , 2019, ACM Trans. Graph..

[64]  Ashok Veeraraghavan,et al.  Ellipsoidal path connections for time-gated rendering , 2019, ACM Trans. Graph..

[65]  Gordon Wetzstein,et al.  Acoustic Non-Line-Of-Sight Imaging , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[66]  Gordon Wetzstein,et al.  Non-line-of-sight Imaging with Partial Occluders and Surface Normals , 2017, ACM Trans. Graph..

[67]  Gordon Wetzstein,et al.  Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations , 2019, NeurIPS.

[68]  Gordon Wetzstein,et al.  DeepVoxels: Learning Persistent 3D Feature Embeddings , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[69]  Felix Heide,et al.  Steady-State Non-Line-Of-Sight Imaging , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[70]  Felix Heide,et al.  Deep-inverse correlography: towards real-time high-resolution non-line-of-sight imaging , 2020, Optica.

[71]  Pratul P. Srinivasan,et al.  NeRF , 2020, ECCV.

[72]  Matthias B. Hullin,et al.  Non-line-of-sight Reconstruction Using Efficient Transient Rendering , 2018, ACM Trans. Graph..

[73]  N KutulakosKiriakos,et al.  Learned feature embeddings for non-line-of-sight imaging and recognition , 2020 .

[74]  Matthias B. Hullin,et al.  Deep Non-Line-of-Sight Reconstruction , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[75]  Felix Heide,et al.  Seeing Around Street Corners: Non-Line-of-Sight Detection and Tracking In-the-Wild Using Doppler Radar , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[76]  Andreas Velten,et al.  Phasor field diffraction based reconstruction for fast non-line-of-sight imaging systems , 2020, Nature Communications.