Computational periscopy with an ordinary digital camera

Computing the amounts of light arriving from different directions enables a diffusely reflecting surface to play the part of a mirror in a periscope—that is, perform non-line-of-sight imaging around an obstruction. Because computational periscopy has so far depended on light-travel distances being proportional to the times of flight, it has mostly been performed with expensive, specialized ultrafast optical systems1–12. Here we introduce a two-dimensional computational periscopy technique that requires only a single photograph captured with an ordinary digital camera. Our technique recovers the position of an opaque object and the scene behind (but not completely obscured by) the object, when both the object and scene are outside the line of sight of the camera, without requiring controlled or time-varying illumination. Such recovery is based on the visible penumbra of the opaque object having a linear dependence on the hidden scene that can be modelled through ray optics. Non-line-of-sight imaging using inexpensive, ubiquitous equipment may have considerable value in monitoring hazardous environments, navigation and detecting hidden adversaries.A faint penumbra in a photograph of a diffuse surface enables recovery of the position of the object creating the penumbra and an image of the scene behind it.

[1]  Vivek K. Goyal,et al.  A Few Photons Among Many: Unmixing Signal and Noise for Photon-Efficient Active Imaging , 2016, IEEE Transactions on Computational Imaging.

[2]  Vivek K. Goyal,et al.  Diffuse Imaging: Creating Optical Images With Unfocused Time-Resolved Illumination and Sensing , 2012, IEEE Signal Processing Letters.

[3]  Gordon Wetzstein,et al.  Non-line-of-sight Imaging with Partial Occluders and Surface Normals , 2017, ACM Trans. Graph..

[4]  James T. Kajiya,et al.  The rendering equation , 1998 .

[5]  J. Bednar,et al.  Alpha-trimmed means and their relationship to median filters , 1984 .

[6]  Antonio Torralba,et al.  Accidental pinhole and pinspeck cameras: Revealing the scene outside the picture , 2012, CVPR.

[7]  R. Raskar,et al.  Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging , 2012, Nature Communications.

[8]  Hui Guo,et al.  Image contrast model of non-line-of-sight imaging based on laser range-gated imaging , 2013 .

[9]  Aongus McCarthy,et al.  Lidar Waveform-Based Analysis of Depth Images Constructed Using Sparse Single-Photon Data , 2015, IEEE Transactions on Image Processing.

[10]  Ramesh Raskar,et al.  Single view reflectance capture using multiplexed scattering and time-of-flight imaging , 2011, SA '11.

[11]  Jaime Martín,et al.  Tracking objects outside the line of sight using 2D intensity images , 2016, Scientific Reports.

[12]  Abderrahim Halimi,et al.  Single-photon three-dimensional imaging at up to 10 kilometers range. , 2017, Optics express.

[13]  Vivek K. Goyal,et al.  Photon-Efficient Computational 3-D and Reflectivity Imaging With Single-Photon Detectors , 2014, IEEE Transactions on Computational Imaging.

[14]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[15]  Martin Laurenzis,et al.  Nonline-of-sight laser gated viewing of scattered photons , 2014 .

[16]  Wolfgang Heidrich,et al.  Diffuse Mirrors: 3D Reconstruction from Diffuse Indirect Illumination Using Inexpensive Time-of-Flight Sensors , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  K. Eliceiri,et al.  Non-line-of-sight imaging using a time-gated single photon avalanche diode. , 2015, Optics express.

[18]  Ashok Veeraraghavan,et al.  Reconstructing rooms using photon echoes: A plane based model and reconstruction algorithm for looking around the corner , 2017, 2017 IEEE International Conference on Computational Photography (ICCP).

[19]  Gordon Wetzstein,et al.  Confocal non-line-of-sight imaging based on the light-cone transform , 2018, Nature.

[20]  Antonio Torralba,et al.  Accidental Pinhole and Pinspeck Cameras , 2014, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Christos Thrampoulidis,et al.  Revealing hidden scenes by photon-efficient occlusion-based opportunistic active imaging. , 2018, Optics express.

[22]  Ramesh Raskar,et al.  Estimating Motion and size of moving non-line-of-sight objects in cluttered environments , 2011, CVPR 2011.

[23]  Ramesh Raskar,et al.  Looking around the corner using transient imaging , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[24]  Ramesh Raskar,et al.  Reconstruction of hidden 3D shapes using diffuse reflections , 2012, Optics express.

[25]  Frédo Durand,et al.  Turning Corners into Cameras: Principles and Methods , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[26]  Vivek K Goyal,et al.  Foundations of Signal Processing , 2014 .

[27]  Kiriakos N. Kutulakos,et al.  The Geometry of First-Returning Photons for Non-Line-of-Sight Imaging , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Vivek K Goyal,et al.  First-Photon Imaging , 2014, Science.

[29]  Gene H. Golub,et al.  Matrix computations (3rd ed.) , 1996 .

[30]  Wolfgang Heidrich,et al.  Low-budget transient imaging using photonic mixer devices , 2013, ACM Trans. Graph..

[31]  ONATHAN,et al.  Non-line-of-sight tracking of people at long range , 2017 .

[32]  Robert Henderson,et al.  Detection and tracking of moving objects hidden from view , 2015, Nature Photonics.

[33]  Martin Laurenzis,et al.  Transient imaging for real-time tracking around a corner , 2016, Security + Defence.

[34]  Jeffrey H. Shapiro,et al.  Exploiting Occlusion in Non-Line-of-Sight Active Imaging , 2017, IEEE Transactions on Computational Imaging.

[35]  Ramesh Raskar,et al.  Occluded Imaging with Time-of-Flight Sensors , 2016, ACM Trans. Graph..

[36]  Frédo Durand,et al.  Inferring Light Fields from Shadows , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.