Optical computing for fast light transport analysis

We present a general framework for analyzing the transport matrix of a real-world scene at full resolution, without capturing many photos. The key idea is to use projectors and cameras to directly acquire eigenvectors and the Krylov subspace of the unknown transport matrix. To do this, we implement Krylov subspace methods partially in optics, by treating the scene as a "black box subroutine" that enables optical computation of arbitrary matrix-vector products. We describe two methods---optical Arnoldi to acquire a low-rank approximation of the transport matrix for relighting; and optical GMRES to invert light transport. Our experiments suggest that good quality relighting and transport inversion are possible from a few dozen low-dynamic range photos, even for scenes with complex shadows, caustics, and other challenging lighting effects.

[1]  S. H. Lee,et al.  Optical Information Processing: Fundamentals , 1981 .

[2]  Lloyd N. Trefethen,et al.  GMRES/CR and Arnoldi/Lanczos as Matrix Approximation Problems , 2018, SIAM J. Sci. Comput..

[3]  Hans-Peter Seidel,et al.  A context-aware light source , 2010, 2010 IEEE International Conference on Computational Photography (ICCP).

[4]  Pierre Ambs,et al.  A short history of optical computing: rise, decline, and evolution , 2009, International Conference on Correlation Optics.

[5]  Pat Hanrahan,et al.  All-frequency shadows using non-linear wavelet lighting approximation , 2003, ACM Trans. Graph..

[6]  Valeria Simoncini,et al.  Theory of Inexact Krylov Subspace Methods and Applications to Scientific Computing , 2003, SIAM J. Sci. Comput..

[7]  Pieter Peers,et al.  Wavelet Environment matting , 2003, Rendering Techniques.

[8]  Richard V. Stone,et al.  Digital optical computer II , 1991 .

[9]  Shree K. Nayar,et al.  Projection defocus analysis for scene capture and image display , 2006, SIGGRAPH 2006.

[10]  Shree K. Nayar,et al.  Multiplexing for Optimal Lighting , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Joaquim Salvi,et al.  Pattern codification strategies in structured light systems , 2004, Pattern Recognit..

[12]  Marc Levoy,et al.  Dual photography , 2005, SIGGRAPH 2005.

[13]  Tian-Tsong Ng,et al.  A Dual Theory of Inverse and Forward Light Transport , 2010, ECCV.

[14]  Michael S. Langer,et al.  When Shadows Become Interreflections , 1999, International Journal of Computer Vision.

[15]  Jörg Liesen,et al.  Convergence analysis of Krylov subspace methods , 2004 .

[16]  Y Fainman,et al.  Optical implementation of an iterative algorithm formatrix inversion. , 1987, Applied optics.

[17]  Pieter Peers,et al.  Inferring reflectance functions from wavelet noise , 2005, EGSR '05.

[18]  Hongyuan Zha,et al.  Low-Rank Matrix Approximation Using the Lanczos Bidiagonalization Process with Applications , 1999, SIAM J. Sci. Comput..

[19]  Ravi Ramamoorthi,et al.  A theory of locally low dimensional light transport , 2007, SIGGRAPH 2007.

[20]  J. Koenderink,et al.  Geometrical modes as a general method to treat diffuse interreflections in radiometry , 1983 .

[21]  Sing H. Lee,et al.  The use of feedback in optical information processing , 1979 .

[22]  Paul E. Debevec,et al.  Acquiring the reflectance field of a human face , 2000, SIGGRAPH.

[23]  Kiriakos N. Kutulakos,et al.  A theory of inverse light transport , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[24]  Pieter Peers,et al.  Compressive light transport sensing , 2009, ACM Trans. Graph..

[25]  Wojciech Matusik,et al.  Acquisition and Rendering of Transparent and Refractive Objects , 2002, Rendering Techniques.

[26]  Tony Q. S. Quek,et al.  Radiometric compensation using stratified inverses , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[27]  R A Athale,et al.  High accuracy computation with linear analog optical systems: a critical study. , 1986, Applied optics.

[28]  Soheil Darabi,et al.  Compressive Dual Photography , 2009, Comput. Graph. Forum.

[29]  D Casasent,et al.  Eigenvector determination by iterative optical methods. , 1981, Applied optics.

[30]  Ronen Basri,et al.  Lambertian Reflectance and Linear Subspaces , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[31]  E. Leith,et al.  The evolution of information optics , 2000, IEEE Journal of Selected Topics in Quantum Electronics.

[32]  Yousef Saad,et al.  Iterative methods for sparse linear systems , 2003 .

[33]  Marc Levoy,et al.  Symmetric photography: exploiting data-sparseness in reflectance fields , 2006, EGSR '06.

[34]  L. Trefethen,et al.  Numerical linear algebra , 1997 .

[35]  Hirokazu Kato,et al.  Real world dynamic appearance enhancement with procam feedback , 2008, PROCAMS '08.

[36]  Hans-Peter Seidel,et al.  Adaptive sampling of reflectance fields , 2007, TOGS.