On Plenoptic Multiplexing and Reconstruction

Photography has been striving to capture an ever increasing amount of visual information in a single image. Digital sensors, however, are limited to recording a small subset of the desired information at each pixel. A common approach to overcoming the limitations of sensing hardware is the optical multiplexing of high-dimensional data into a photograph. While this is a well-studied topic for imaging with color filter arrays, we develop a mathematical framework that generalizes multiplexed imaging to all dimensions of the plenoptic function. This framework unifies a wide variety of existing approaches to analyze and reconstruct multiplexed data in either the spatial or the frequency domain. We demonstrate many practical applications of our framework including high-quality light field reconstruction, the first comparative noise analysis of light field attenuation masks, and an analysis of aliasing in multiplexing applications.

[1]  Andrew Lumsdaine,et al.  Rich image capture with plenoptic cameras , 2010, 2010 IEEE International Conference on Computational Photography (ICCP).

[2]  Richard G. Baraniuk,et al.  A new compressive imaging camera architecture using optical-domain compression , 2006, Electronic Imaging.

[3]  Shree K. Nayar,et al.  Video super-resolution using controlled subpixel detector shifts , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Frédo Durand,et al.  4D frequency analysis of computational cameras for depth of field extension , 2009, SIGGRAPH '09.

[5]  M. Levoy,et al.  Light field microscopy , 2006, SIGGRAPH 2006.

[6]  Shree K. Nayar,et al.  Generalized Mosaicing : High Dynamic Range in a Wide Field of View 247 , 2001 .

[7]  Ren Ng Fourier Slice Photography , 2005 .

[8]  A. Wuttig Optimal transformations for optical multiplex measurements in the presence of photon noise. , 2005, Applied optics.

[9]  Peter Kohl,et al.  Pixel multiplexing for high-speed multi-resolution fluorescence imaging , 2009 .

[10]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH.

[11]  J. Rinehart,et al.  U . S . Patent , 2006 .

[12]  Ramesh Raskar,et al.  Non-refractive modulators for encoding and capturing scene appearance and depth , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[13]  Narendra Ahuja,et al.  Split Aperture Imaging for High Dynamic Range , 2004, International Journal of Computer Vision.

[14]  Gordon Wetzstein,et al.  Computational Plenoptic Imaging , 2011, Comput. Graph. Forum.

[15]  Gordon Wetzstein,et al.  Sensor saturation in Fourier multiplexed imaging , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[16]  Rama Chellappa,et al.  P2C2: Programmable pixel compressive camera for high speed imaging , 2011, CVPR 2011.

[17]  Lei Zhang,et al.  Image demosaicing: a systematic survey , 2008, Electronic Imaging.

[18]  W. Freeman,et al.  Understanding Camera Trade-Offs through a Bayesian Analysis of Light Field Projections , 2008, ECCV.

[19]  Sabine Süsstrunk,et al.  Linear demosaicing inspired by the human visual system , 2005, IEEE Transactions on Image Processing.

[20]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[21]  Shree K. Nayar,et al.  Generalized Assorted Pixel Camera: Postcapture Control of Resolution, Dynamic Range, and Spectrum , 2010, IEEE Transactions on Image Processing.

[22]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[23]  Gil Bub,et al.  Pixel Multiplexing for Simultaneous High Resolution High Speed Image Capture , 2010 .

[24]  Shree K. Nayar,et al.  Optical Splitting Trees for High-Precision Monocular Imaging , 2007, IEEE Computer Graphics and Applications.

[25]  Andrew Lumsdaine,et al.  The focused plenoptic camera , 2009, 2009 IEEE International Conference on Computational Photography (ICCP).

[26]  Ramesh Raskar,et al.  Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing , 2007, SIGGRAPH 2007.

[27]  Ashok Veeraraghavan,et al.  Optimal coded sampling for temporal super-resolution , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[28]  Edward H. Adelson,et al.  Single Lens Stereo with a Plenoptic Camera , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  E E Fenimore,et al.  New family of binary arrays for coded aperture imaging. , 1989, Applied optics.

[30]  Shree K. Nayar,et al.  Multiplexing for Optimal Lighting , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Wolfgang Heidrich,et al.  The Design of an Inexpensive Very High Resolution Scan Camera System , 2004, Comput. Graph. Forum.

[32]  Andrew Lumsdaine,et al.  Unified Frequency Domain Analysis of Lightfield Cameras , 2008, ECCV.

[33]  E. Adelson,et al.  The Plenoptic Function and the Elements of Early Vision , 1991 .

[34]  Ramesh Raskar,et al.  Coded Strobing Photography: Compressive Sensing of High Speed Periodic Videos , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Marc Levoy,et al.  High performance imaging using large camera arrays , 2005, SIGGRAPH 2005.

[36]  Anat Levin,et al.  4D frequency analysis of computational cameras for depth of field extension , 2009, SIGGRAPH '09.

[37]  Shree K. Nayar,et al.  Enhancing resolution along multiple imaging dimensions using assorted pixels , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[38]  J. Greivenkamp Color dependent optical prefilter for the suppression of aliasing artifacts. , 1990, Applied optics.

[39]  M. Landy,et al.  The Plenoptic Function and the Elements of Early Vision , 1991 .

[40]  Tomoyuki Nishita,et al.  Extracting depth and matte using a color-filtered aperture , 2008, SIGGRAPH Asia '08.

[41]  Shree K. Nayar,et al.  Radiometric self calibration , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[42]  Ramesh Raskar,et al.  Reinterpretable Imager: Towards Variable Post‐Capture Space, Angle and Time Resolution in Photography , 2010, Comput. Graph. Forum.

[43]  Douglas Lanman,et al.  Shield fields: modeling and capturing 3D occluders , 2008, SIGGRAPH 2008.

[44]  Ashok Veeraraghavan,et al.  Flexible Voxels for Motion-Aware Videography , 2010, ECCV.

[45]  Frédo Durand,et al.  Linear view synthesis using a dimensionality gap light field prior , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[46]  Ramesh Raskar,et al.  Switchable primaries using shiftable layers of color filter arrays , 2011, SIGGRAPH 2011.

[47]  Ravindra Athale,et al.  Flexible multimodal camera using a light field architecture , 2009, 2009 IEEE International Conference on Computational Photography (ICCP).

[48]  Gordon Wetzstein,et al.  Computational Plenoptic Imaging , 2011, SIGGRAPH '12.

[49]  Andrew Gardner,et al.  Performance relighting and reflectance transformation with time-multiplexed illumination , 2005, ACM Trans. Graph..

[50]  Tom E. Bishop,et al.  Light field superresolution , 2009, 2009 IEEE International Conference on Computational Photography (ICCP).