Improving resolution and depth-of-field of light field cameras using a hybrid imaging system

Current light field (LF) cameras provide low spatial resolution and limited depth-of-field (DOF) control when compared to traditional digital SLR (DSLR) cameras. We show that a hybrid imaging system consisting of a standard LF camera and a high-resolution standard camera enables (a) achieve high-resolution digital refocusing, (b) better DOF control than LF cameras, and (c) render graceful high-resolution viewpoint variations, all of which were previously unachievable. We propose a simple patch-based algorithm to super-resolve the low-resolution views of the light field using the high-resolution patches captured using a high-resolution SLR camera. The algorithm does not require the LF camera and the DSLR to be co-located or for any calibration information regarding the two imaging systems. We build an example prototype using a Lytro camera (380×380 pixel spatial resolution) and a 18 megapixel (MP) Canon DSLR camera to generate a light field with 11 MP resolution (9× super-resolution) and about 1 over 9 th of the DOF of the Lytro camera. We show several experimental results on challenging scenes containing occlusions, specularities and complex non-lambertian materials, demonstrating the effectiveness of our approach.

[1]  Harry Shum,et al.  Image completion with structure propagation , 2005, ACM Trans. Graph..

[2]  Pieter Peers,et al.  SubEdit: a representation for editing measured heterogeneous subsurface scattering , 2009, SIGGRAPH 2009.

[3]  Jean-Michel Morel,et al.  A non-local algorithm for image denoising , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[4]  Ashok Veeraraghavan,et al.  Towards Motion Aware Light Field Video for Dynamic Scenes , 2013, 2013 IEEE International Conference on Computer Vision.

[5]  Seungyong Lee,et al.  Video deblurring for hand-held cameras using patch-based synthesis , 2012, ACM Trans. Graph..

[6]  Yasuyuki Matsushita,et al.  High-resolution hyperspectral imaging via matrix factorization , 2011, CVPR 2011.

[7]  Hong Chang,et al.  Super-resolution through neighbor embedding , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[8]  R. Shack,et al.  History and principles of Shack-Hartmann wavefront sensing. , 2001, Journal of refractive surgery.

[9]  Ashok Veeraraghavan,et al.  Light field denoising, light field superresolution and stereo camera based refocussing using a GMM light field patch prior , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[10]  Ramesh Raskar,et al.  Reinterpretable Imager: Towards Variable Post‐Capture Space, Angle and Time Resolution in Photography , 2010, Comput. Graph. Forum.

[11]  Raanan Fattal,et al.  Image and video upscaling from local self-examples , 2011, TOGS.

[12]  Alexei A. Efros,et al.  Image quilting for texture synthesis and transfer , 2001, SIGGRAPH.

[13]  A. Lumsdaine,et al.  Superresolution with Plenoptic Camera 2 . 0 , 2009 .

[14]  Chia-Kai Liang,et al.  Programmable aperture photography: multiplexed light field acquisition , 2008, SIGGRAPH 2008.

[15]  Stephen Lin,et al.  High resolution multispectral video capture with a hybrid camera system , 2011, CVPR 2011.

[16]  Tom E. Bishop,et al.  Light field superresolution , 2009, 2009 IEEE International Conference on Computational Photography (ICCP).

[17]  Gordon Wetzstein,et al.  Compressive light field photography using overcomplete dictionaries and optimized projections , 2013, ACM Trans. Graph..

[18]  P. Hanrahan,et al.  Light Field Photography with a Hand-held Plenoptic Camera , 2005 .

[19]  Adam Finkelstein,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, SIGGRAPH 2009.

[20]  Andrew Lumsdaine,et al.  Superresolution with Plenoptic 2.0 Cameras , 2009 .

[21]  Marc Levoy,et al.  High performance imaging using large camera arrays , 2005, ACM Trans. Graph..

[22]  Marc Levoy,et al.  Synthetic aperture confocal imaging , 2004, ACM Trans. Graph..

[23]  Aggelos K. Katsaggelos,et al.  Compressive Light Field Sensing , 2012, IEEE Transactions on Image Processing.

[24]  Jitendra Malik,et al.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[25]  Albert Wang,et al.  An angle-sensitive CMOS imager for single-sensor 3D photography , 2011, 2011 IEEE International Solid-State Circuits Conference.

[26]  Ramesh Raskar,et al.  Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing , 2007, ACM Trans. Graph..

[27]  Zhengyou Zhang,et al.  Virtual View Generation with a Hybrid Camera Array , 2009 .

[28]  Michal Irani,et al.  Internal statistics of a single natural image , 2011, CVPR 2011.

[29]  E. Adelson,et al.  The Plenoptic Function and the Elements of Early Vision , 1991 .

[30]  Sven Wanner,et al.  Variational Light Field Analysis for Disparity Estimation and Super-Resolution , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[32]  Shree K. Nayar,et al.  Motion deblurring using hybrid imaging , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[33]  Keith J. Hanna,et al.  Hybrid stereo camera: an IBR approach for synthesis of very high resolution stereoscopic image sequences , 2001, SIGGRAPH.

[34]  David Salesin,et al.  Spatio-angular resolution tradeoffs in integral photography , 2006, EGSR '06.

[35]  Yu-Wing Tai,et al.  Modeling the Calibration Pipeline of the Lytro Camera for High Quality Light-Field Image Reconstruction , 2013, 2013 IEEE International Conference on Computer Vision.

[36]  David G. Lowe,et al.  Fast Approximate Nearest Neighbors with Automatic Algorithm Configuration , 2009, VISAPP.

[37]  J. Fleischer,et al.  High-Resolution Light-Field Microscopy , 2013 .

[38]  Stefan B. Williams,et al.  Decoding, Calibration and Rectification for Lenselet-Based Plenoptic Cameras , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[39]  William T. Freeman,et al.  Example-Based Super-Resolution , 2002, IEEE Computer Graphics and Applications.

[40]  Thomas S. Huang,et al.  Image Super-Resolution Via Sparse Representation , 2010, IEEE Transactions on Image Processing.