Inferring Light Fields from Shadows

We present a method for inferring a 4D light field of a hidden scene from 2D shadows cast by a known occluder on a diffuse wall. We do this by determining how light naturally reflected off surfaces in the hidden scene interacts with the occluder. By modeling the light transport as a linear system, and incorporating prior knowledge about light field structures, we can invert the system to recover the hidden scene. We demonstrate results of our inference method across simulations and experiments with different types of occluders. For instance, using the shadow cast by a real house plant, we are able to recover low resolution light fields with different levels of texture and parallax complexity. We provide two experimental results: a human subject and two planar elements at different depths.

[1]  Edward H. Adelson,et al.  Single Lens Stereo with a Plenoptic Camera , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  Frédo Durand,et al.  Image and depth from a conventional camera with a coded aperture , 2007, ACM Trans. Graph..

[3]  Ashley Tews,et al.  Pedestrian detection in industrial environments: Seeing around corners , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[4]  Vivek K. Goyal,et al.  Photon-Efficient Computational 3-D and Reflectivity Imaging With Single-Photon Detectors , 2014, IEEE Transactions on Computational Imaging.

[5]  Wei-Chao Chen,et al.  Light field mapping: efficient representation and hardware rendering of surface light fields , 2002, SIGGRAPH.

[6]  Robert Henderson,et al.  Detection and tracking of moving objects hidden from view , 2015, Nature Photonics.

[7]  Ramesh Raskar,et al.  Looking around the corner using transient imaging , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[8]  Ramesh Raskar,et al.  Estimating Motion and size of moving non-line-of-sight objects in cluttered environments , 2011, CVPR 2011.

[9]  Adam L. Cohen.  Anti-pinhole Imaging , 1982 .

[10]  David Salesin,et al.  Surface light fields for 3D photography , 2000, SIGGRAPH.

[11]  P. Hanrahan,et al.  Light Field Photography with a Hand-held Plenoptic Camera , 2005 .

[12]  Stephen Lin,et al.  Coded Aperture Pairs for Depth from Defocus and Defocus Deblurring , 2011, International Journal of Computer Vision.

[13]  Ravi Ramamoorthi,et al.  Learning to Synthesize a 4D RGBD Light Field from a Single Image , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[14]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[15]  Jake K. Aggarwal,et al.  Human detection using depth information by Kinect , 2011, CVPR 2011 WORKSHOPS.

[16]  Jeffrey H. Shapiro,et al.  Exploiting Occlusion in Non-Line-of-Sight Active Imaging , 2017, IEEE Transactions on Computational Imaging.

[17]  R. Raskar,et al.  Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging , 2012, Nature Communications.

[18]  Frédo Durand,et al.  Frequency analysis and sheared filtering for shadow light fields of complex occluders , 2011, TOGS.

[19]  Ramesh Raskar,et al.  Occluded Imaging with Time-of-Flight Sensors , 2016, ACM Trans. Graph..

[20]  Frédo Durand,et al.  Image and depth from a conventional camera with a coded aperture , 2007, SIGGRAPH 2007.

[21]  Harry Shum,et al.  Plenoptic sampling , 2000, SIGGRAPH.

[22]  Frédo Durand,et al.  Turning Corners into Cameras: Principles and Methods , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[23]  Frédo Durand,et al.  Linear view synthesis using a dimensionality gap light field prior , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[24]  Kiriakos N. Kutulakos,et al.  The Geometry of First-Returning Photons for Non-Line-of-Sight Imaging , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Antonio Torralba,et al.  Accidental Pinhole and Pinspeck Cameras , 2014, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Vivek K. Goyal,et al.  Photon-efficient computational imaging with a single-photon camera , 2016 .

[27]  Leonard McMillan,et al.  Dynamically reparameterized light fields , 2000, SIGGRAPH.

[28]  Marc Levoy,et al.  High performance imaging using large camera arrays , 2005, SIGGRAPH 2005.

[29]  G. J. Burton,et al.  Color and spatial structure in natural scenes. , 1987, Applied optics.

[30]  Gordon Wetzstein,et al.  Computational imaging with multi-camera time-of-flight systems , 2016, ACM Trans. Graph..

[31]  Martin Laurenzis,et al.  Dual-mode optical sensing: three-dimensional imaging and seeing around a corner , 2016 .