Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing

We describe a theoretical framework for reversibly modulating 4D light fields using an attenuating mask in the optical path of a lens based camera. Based on this framework, we present a novel design to reconstruct the 4D light field from a 2D camera image without any additional refractive elements as required by previous light field cameras. The patterned mask attenuates light rays inside the camera instead of bending them, and the attenuation recoverably encodes the rays on the 2D sensor. Our mask-equipped camera focuses just as a traditional camera to capture conventional 2D photos at full sensor resolution, but the raw pixel values also hold a modulated 4D light field. The light field can be recovered by rearranging the tiles of the 2D Fourier transform of sensor values into 4D planes, and computing the inverse Fourier transform. In addition, one can also recover the full resolution image information for the in-focus parts of the scene. We also show how a broadband mask placed at the lens enables us to compute refocused images at full sensor resolution for layered Lambertian scenes. This partial encoding of 4D ray-space data enables editing of image contents by depth, yet does not require computational recovery of the complete 4D light field.

[1]  William H. Richardson,et al.  Bayesian-Based Iterative Method of Image Restoration , 1972 .

[2]  R. Accorsi,et al.  Optimal coded aperture patterns for improved SNR in nuclear medicine imaging , 2001 .

[3]  Marc Levoy,et al.  Using plane + parallax for calibrating dense camera arrays , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[4]  Shree K. Nayar,et al.  Lensless Imaging with a Controllable Aperture , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[5]  Edward R. Dowski,et al.  Wavefront coding: a modern method of achieving high-performance and/or low-cost imaging systems , 1999, Optics & Photonics.

[6]  Jason C. Yang,et al.  A light field camera for image based rendering , 2000 .

[7]  Bahram Javidi,et al.  Three-Dimensional Television, Video and Display Technology , 2002 .

[8]  E E Fenimore,et al.  New family of binary arrays for coded aperture imaging. , 1989, Applied optics.

[9]  Herbert E. Ives,et al.  A Camera for Making Parallax Panoramagrams , 1928 .

[10]  Paul Haeberli.  A Multifocus Method for Controlling Depth of Field , 2005 .

[11]  G E Johnson,et al.  Passive ranging through wave-front coding: information and application. , 2000, Applied optics.

[12]  L. Lucy.  An iterative technique for the rectification of observed distributions , 1974 .

[13]  F. Okano,et al.  Three-dimensional video system based on integral photography , 1999 .

[14]  J van der Gracht,et al.  Broadband behavior of an optical-digital focus-invariant system. , 1996, Optics letters.

[15]  R. Fergus,et al.  Random Lens Imaging , 2006 .

[16]  Olga Veksler,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  Ren Ng.  Fourier Slice Photography , 2005 .

[18]  Edward H. Adelson,et al.  Single Lens Stereo with a Plenoptic Camera , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  Wolfgang Heidrich,et al.  The Design of an Inexpensive Very High Resolution Scan Camera System , 2004, Comput. Graph. Forum.

[20]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[21]  J Arai,et al.  Real-time pickup method for a three-dimensional image based on integral photography. , 1997, Applied optics.

[22]  Eero P. Simoncelli,et al.  Range estimation by optical differentiation. , 1998, Journal of the Optical Society of America. A, Optics, image science, and vision.

[23]  Ramesh Raskar,et al.  Coded exposure photography: motion deblurring using fluttered shutter , 2006, SIGGRAPH 2006.

[24]  John Hart,et al.  ACM Transactions on Graphics , 2004, SIGGRAPH 2004.

[25]  Marc Levoy,et al.  Synthetic aperture confocal imaging , 2004, SIGGRAPH 2004.

[26]  Takashi Matsuyama,et al.  Depth measurement by the multi-focus camera , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[27]  R. A. Fessenden.  Wireless Telephony , 1919, Nature.

[28]  Shree K. Nayar,et al.  Programmable Imaging: Towards a Flexible Camera , 2006, International Journal of Computer Vision.

[29]  David Salesin,et al.  Spatio-angular resolution tradeoffs in integral photography , 2006, EGSR '06.

[30]  B. Javidi,et al.  Integral imaging with improved depth of field by use of amplitude-modulated microlens arrays. , 2004, Applied optics.

[31]  G. Skinner.  X-ray imaging with coded masks. , 1988, Scientific American.

[32]  W. Cathey,et al.  Extended depth of field through wave-front coding. , 1995, Applied optics.

[33]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[34]  Subhasis Chaudhuri,et al.  Depth From Defocus: A Real Aperture Imaging Approach , 1999, Springer New York.

[35]  Ramesh Raskar,et al.  Non-refractive modulators for encoding and capturing scene appearance and depth , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  Leonard McMillan,et al.  Dynamically reparameterized light fields , 2000, SIGGRAPH.

[37]  Marc Levoy,et al.  High performance imaging using large camera arrays , 2005, SIGGRAPH 2005.

[38]  Wenyang Sun,et al.  Rainbow volume holographic imaging , 2004, Conference on Lasers and Electro-Optics, 2004. (CLEO)..

[39]  Alan V. Oppenheim,et al.  Discrete-time Signal Processing. Vol.2 , 2001 .

[40]  David J. Field,et al.  What Is the Goal of Sensory Coding? , 1994, Neural Computation.

[41]  G. Lippmann.  Epreuves reversibles donnant la sensation du relief , 1908 .

[42]  David Salesin,et al.  Interactive digital photomontage , 2004, SIGGRAPH 2004.

[43]  Ramesh Raskar,et al.  Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing , 2007, SIGGRAPH 2007.

[44]  Shree K. Nayar,et al.  High dynamic range imaging: spatially varying pixel exposures , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[45]  P. Hanrahan,et al.  Light Field Photography with a Hand-held Plenoptic Camera , 2005 .