Phasor field diffraction based reconstruction for fast non-line-of-sight imaging systems

Non-line-of-sight (NLOS) imaging recovers objects using diffusely reflected indirect light using transient illumination devices in combination with a computational inverse method. While capture systems capable of collecting light from the entire NLOS relay surface can be much more light efficient than single pixel point scanning detection, current reconstruction algorithms for such systems have computational and memory requirements that prevent real-time NLOS imaging. Existing real-time demonstrations also use retroreflective targets and reconstruct at resolutions far below the hardware limits. Our method presented here enables the reconstruction of room-sized scenes from non-confocal, parallel multi-pixel measurements in seconds with less memory usage. We anticipate that our method will enable real-time NLOS imaging when used with emerging single-photon avalanche diode array detectors with resolution only limited by the temporal resolution of the sensor. Current implementations of non-line-of-sight imaging use reconstruction algorithms that are difficult to implement fast enough for real-time application using light efficient equipment. The authors present an algorithm for non-line-of-sight imaging that is low complexity and allows fast and efficient reconstruction on a standard computer.

[1]  Anbo Wang,et al.  Fast-Fourier-transform based numerical integration method for the Rayleigh-Sommerfeld diffraction formula. , 2006, Applied optics.

[2]  Xiaobin Wang,et al.  Improved algorithm of non-line-of-sight imaging based on the Bayesian statistics. , 2019, Journal of the Optical Society of America. A, Optics, image science, and vision.

[3]  Felix Heide,et al.  Steady-State Non-Line-Of-Sight Imaging , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Gordon Wetzstein,et al.  Acoustic Non-Line-Of-Sight Imaging , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Christos Thrampoulidis,et al.  Using Unknown Occluders to Recover Hidden Scenes , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Reinhard Klein,et al.  Solving trigonometric moment problems for fast transient imaging , 2015, ACM Trans. Graph..

[7]  Federica Villa,et al.  3D RGB Non-Line-Of-Sight single-pixel imaging , 2019, Imaging and Applied Optics 2019 (COSI, IS, MATH, pcAOP).

[8]  M. Cecchini,et al.  Ultrastructural Characterization of the Lower Motor System in a Mouse Model of Krabbe Disease , 2016, Scientific Reports.

[9]  Andreas Velten,et al.  Error Backprojection Algorithms for Non-Line-of-Sight Imaging , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Gordon Wetzstein,et al.  A dataset for benchmarking time-resolved non-line-of-sight imaging , 2019, SIGGRAPH Posters.

[11]  Jeffrey H. Shapiro,et al.  Exploiting Occlusion in Non-Line-of-Sight Active Imaging , 2017, IEEE Transactions on Computational Imaging.

[12]  Kiriakos N. Kutulakos,et al.  A Theory of Fermat Paths for Non-Line-Of-Sight Shape Reconstruction , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Diego Gutierrez,et al.  Fast back-projection for non-line of sight reconstruction , 2017, SIGGRAPH Posters.

[14]  Jaime Martín,et al.  Tracking objects outside the line of sight using 2D intensity images , 2016, Scientific Reports.

[15]  James Davis,et al.  5d time-light transport matrix: What can we reason about scene properties? , 2008 .

[16]  Kiriakos N. Kutulakos,et al.  The Geometry of First-Returning Photons for Non-Line-of-Sight Imaging , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Andreas Velten,et al.  Analysis of Feature Visibility in Non-Line-Of-Sight Measurements , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Gordon Wetzstein,et al.  Non-line-of-sight Imaging with Partial Occluders and Surface Normals , 2017, ACM Trans. Graph..

[19]  Yusuke Murayama,et al.  Unravelling cerebellar pathways with high temporal precision targeting motor and extensive sensory and parietal networks , 2012, Nature Communications.

[20]  Ramesh Raskar,et al.  Looking Around the Corner using Ultrafast Transient Imaging , 2011, International Journal of Computer Vision.

[21]  V. Nascov,et al.  Fast computation algorithm for the Rayleigh-Sommerfeld diffraction formula using a type of scaled convolution. , 2009, Applied optics.

[22]  Gary F. Margrave,et al.  Numerical Methods of Exploration Seismology , 2019 .

[23]  Charles Saunders,et al.  Computational periscopy with an ordinary digital camera , 2019, Nature.

[24]  Syed Azer Reza,et al.  Phasor field waves: A Huygens-like light transport model for non-line-of-sight imaging applications. , 2018, Optics express.

[25]  Gordon Wetzstein,et al.  Confocal non-line-of-sight imaging based on the light-cone transform , 2018, Nature.

[26]  Ramesh Raskar,et al.  Thermal Non-Line-of-Sight Imaging , 2019, 2019 IEEE International Conference on Computational Photography (ICCP).

[27]  Qionghai Dai,et al.  Frequency Analysis of Transient Light Transport with Applications in Bare Sensor Imaging , 2012, ECCV.

[28]  Aswin C. Sankaranarayanan,et al.  Convolutional Approximations to the General Non-Line-of-Sight Imaging Operator , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[29]  Federica Villa,et al.  Non-Line-of-Sight Three-Dimensional Imaging with a Single-Pixel Camera , 2019, Physical Review Applied.

[30]  Frédo Durand,et al.  Inferring Light Fields from Shadows , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[31]  Jeremy A. Teichman,et al.  Phasor field waves: a mathematical treatment. , 2019, Optics express.

[32]  Gary F. Margrave,et al.  Direct Fourier migration for vertical velocity variations , 2001 .

[33]  Diego Gutierrez,et al.  Non-line-of-sight imaging using phasor-field virtual wave optics , 2018, Nature.

[34]  Christos Thrampoulidis,et al.  Revealing hidden scenes by photon-efficient occlusion-based opportunistic active imaging. , 2018, Optics express.

[35]  Ramesh Raskar,et al.  Reconstruction of hidden 3D shapes using diffuse reflections , 2012, Optics express.

[36]  Ashok Veeraraghavan,et al.  SNLOS: Non-line-of-sight Scanning through Temporal Focusing , 2019, 2019 IEEE International Conference on Computational Photography (ICCP).

[37]  R. Raskar,et al.  Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging , 2012, Nature Communications.

[38]  Andreas Velten,et al.  Feature selection and back-projection algorithms for nonline-of-sight laser–gated viewing , 2014, J. Electronic Imaging.

[39]  K. Eliceiri,et al.  Non-line-of-sight imaging using a time-gated single photon avalanche diode. , 2015, Optics express.

[40]  Syed Azer Reza,et al.  Phasor field waves: experimental demonstrations of wave-like properties. , 2019, Optics express.

[41]  Jeffrey H. Shapiro,et al.  Paraxial theory of phasor-field imaging. , 2019, Optics express.

[42]  Gordon Wetzstein,et al.  Wave-based non-line-of-sight imaging using fast f-k migration , 2019, ACM Trans. Graph..

[43]  Wolfgang Heidrich,et al.  Diffuse Mirrors: 3D Reconstruction from Diffuse Indirect Illumination Using Inexpensive Time-of-Flight Sensors , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[44]  R. Plevin,et al.  MAP kinase phosphatase 2 deficient mice develop attenuated experimental autoimmune encephalomyelitis through regulating dendritic cells and T cells , 2016, Scientific Reports.

[45]  Sehoon Ha,et al.  Iterative Training of Dynamic Skills Inspired by Human Coaching Techniques , 2014, ACM Trans. Graph..

[46]  P. Drummond,et al.  Time reversed acoustics , 1997 .

[47]  Aswin C. Sankaranarayanan,et al.  Beyond Volumetric Albedo — A Surface Optimization Framework for Non-Line-Of-Sight Imaging , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Jaakko Astola,et al.  Advances in Signal Transforms: Theory and Applications , 2007 .