Tensor photography : exploring space of 4D modulations inside traditional camera designs

Light field photography has gained a significant research in the last two decades: today, commercial light field cameras are widely available demonstrating capabilities such as post-capture refocus, 3D photography and view point changes. But, most traditional acquisition approaches either multiplex a low resolution light field into a single sensor image or require multiple photographs to be taken for acquiring high resolution light field. In this thesis, we design, implement and analyze a new light field camera architecture that allows capture and reconstruction of higher resolution light fields in a single shot. The proposed architecture comprises three key components: light field atoms as sparse representation of natural light fields, an optical design to allow capture of optimized 2D light field projections and robust sparse reconstruction methods to recover a 4D light field from a single coded 2D projection. In addition we also explore other applications including compressive focal stack reconstructions, light field compression and denoising. Thesis Supervisor: Ramesh Raskar Title: Associate Professor Program in Media Arts and Sciences

[1]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[2]  Guillermo Sapiro,et al.  Online dictionary learning for sparse coding , 2009, ICML '09.

[3]  Gordon Wetzstein,et al.  Adaptive image synthesis for compressive displays , 2013, ACM Trans. Graph..

[4]  Edmund Y. Lam,et al.  A high-resolution lightfield camera with dual-mask design , 2012, Other Conferences.

[5]  Emmanuel J. Candès,et al.  NESTA: A Fast and Accurate First-Order Method for Sparse Recovery , 2009, SIAM J. Imaging Sci..

[6]  Andrew Lumsdaine,et al.  The focused plenoptic camera , 2009, 2009 IEEE International Conference on Computational Photography (ICCP).

[7]  Mark A. Neifeld,et al.  Compressive light field imaging , 2010, Defense + Commercial Sensing.

[8]  Gordon Wetzstein,et al.  Tensor displays , 2012, ACM Trans. Graph..

[9]  Chia-Kai Liang,et al.  Programmable aperture photography: multiplexed light field acquisition , 2008, SIGGRAPH 2008.

[10]  Pierre Vandergheynst,et al.  Light field compressive sensing in camera arrays , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[11]  Stephen P. Boyd,et al.  Enhancing Sparsity by Reweighted ℓ1 Minimization , 2007, 0711.1612.

[12]  Gordon Wetzstein,et al.  Polarization fields: dynamic light field display using multi-layer LCDs , 2011, SA '11.

[13]  E.J. Candes,et al.  An Introduction To Compressive Sampling , 2008, IEEE Signal Processing Magazine.

[14]  Dan Feldman.  Coresets and Their Applications , 2012 .

[15]  Ting Sun,et al.  Single-pixel imaging via compressive sampling , 2008, IEEE Signal Process. Mag..

[16]  Douglas Lanman,et al.  Shield fields: modeling and capturing 3D occluders , 2008, SIGGRAPH 2008.

[17]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[18]  Ramesh Raskar,et al.  Lens Aberration Correction Using Locally Optimal Mask Based Low Cost Light Field Cameras , 2010 .

[19]  Dan Feldman,et al.  From High Definition Image to Low Space Optimization , 2011, SSVM.

[20]  Marc Levoy,et al.  High performance imaging using large camera arrays , 2005, SIGGRAPH 2005.

[21]  E. Candès,et al.  Stable signal recovery from incomplete and inaccurate measurements , 2005, math/0503066.

[22]  Tom E. Bishop,et al.  Light field superresolution , 2009, 2009 IEEE International Conference on Computational Photography (ICCP).

[23]  David L Donoho,et al.  Compressed sensing , 2006, IEEE Transactions on Information Theory.

[24]  Allen Y. Yang,et al.  Fast ℓ1-minimization algorithms and an application in robust face recognition: A review , 2010, 2010 IEEE International Conference on Image Processing.

[25]  Thomas S. Huang,et al.  Coupled Dictionary Training for Image Super-Resolution , 2012, IEEE Transactions on Image Processing.

[26]  Michael Elad,et al.  On Single Image Scale-Up Using Sparse-Representations , 2010, Curves and Surfaces.

[27]  Gordon Wetzstein,et al.  On Plenoptic Multiplexing and Reconstruction , 2012, International Journal of Computer Vision.

[28]  Pankaj K. Agarwal,et al.  Approximating extent measures of points , 2004, JACM.

[29]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[30]  Shree K. Nayar,et al.  Video from a single coded exposure photograph using a learned over-complete dictionary , 2011, 2011 International Conference on Computer Vision.

[31]  Guillermo Sapiro,et al.  Online Learning for Matrix Factorization and Sparse Coding , 2009, J. Mach. Learn. Res..

[32]  Rama Chellappa,et al.  P2C2: Programmable pixel compressive camera for high speed imaging , 2011, CVPR 2011.

[33]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[34]  Frédo Durand,et al.  4D frequency analysis of computational cameras for depth of field extension , 2009, SIGGRAPH '09.

[35]  David Salesin,et al.  Spatio-angular resolution tradeoffs in integral photography , 2006, EGSR '06.

[36]  D. Donoho.  For most large underdetermined systems of linear equations the minimal 𝓁1‐norm solution is also the sparsest solution , 2006 .

[37]  Yonina C. Eldar,et al.  Compressed Sensing with Coherent and Redundant Dictionaries , 2010, ArXiv.

[38]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[39]  Ren Ng.  Fourier Slice Photography , 2005 .

[40]  Michael P. Friedlander,et al.  Probing the Pareto Frontier for Basis Pursuit Solutions , 2008, SIAM J. Sci. Comput..

[41]  P. Hanrahan,et al.  Light Field Photography with a Hand-held Plenoptic Camera , 2005 .

[42]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[43]  Michael Elad,et al.  Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries , 2006, IEEE Transactions on Image Processing.

[44]  Guillermo Sapiro,et al.  Learning to Sense Sparse Signals: Simultaneous Sensing Matrix and Sparsifying Dictionary Optimization , 2009, IEEE Transactions on Image Processing.

[45]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[46]  Aggelos K. Katsaggelos,et al.  Compressive Light Field Sensing , 2012, IEEE Transactions on Image Processing.

[47]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[48]  Joel A. Tropp,et al.  Signal Recovery From Random Measurements Via Orthogonal Matching Pursuit , 2007, IEEE Transactions on Information Theory.

[49]  Gordon Wetzstein,et al.  Layered 3D: tomographic image synthesis for attenuation-based light field and high dynamic range displays , 2011, SIGGRAPH 2011.

[50]  Ramesh Raskar,et al.  Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing , 2007, SIGGRAPH 2007.

[51]  Joel A. Tropp,et al.  Topics in sparse approximation , 2004 .

[52]  Edward H. Adelson,et al.  Single Lens Stereo with a Plenoptic Camera , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[53]  Michael B. Wakin,et al.  A geometric approach to multi-view compressive imaging , 2012, EURASIP J. Adv. Signal Process..

[54]  Lennart Wietzke,et al.  Single lens 3D-camera with extended depth-of-field , 2012, Electronic Imaging.

[55]  Michael Elad,et al.  Multiscale Sparse Image Representation with Learned Dictionaries (PREPRINT) , 2007 .

[56]  R. Fergus,et al.  Random Lens Imaging , 2006 .

[57]  Balas K. Natarajan,et al.  Sparse Approximate Solutions to Linear Systems , 1995, SIAM J. Comput..

[58]  Michael Elad,et al.  K-SVD and its non-negative variant for dictionary design , 2005, SPIE Optics + Photonics.

[59]  W. Freeman,et al.  Understanding Camera Trade-Offs through a Bayesian Analysis of Light Field Projections , 2008, ECCV.

[60]  Stephen J. Wright,et al.  Computational Methods for Sparse Solution of Linear Inverse Problems , 2010, Proceedings of the IEEE.