Super-Resolved Fourier-Slice Refocusing in Plenoptic Cameras

Plenoptic cameras are a new type of sensors that capture the four-dimensional lightfield of a scene. Processing the recorded lightfield, these cameras extend the capabilities of current commercial cameras offering the possibility of focusing the scene after the shot or obtaining 3D information. Conventional photographs focused on certain planes can be obtained through projections of the four-dimensional lightfield onto two spatial dimensions. These photographs can be efficiently computed using the Fourier Slice technique, but their resolution is limited since a plenoptic camera trades off spatial resolution for angular resolution. In order to remove this limitation, several super-resolution methods have been recently developed to increase the spatial resolution of plenoptic cameras. In this paper, we study the super-resolution problem in plenoptic cameras and show how to efficiently compute super-resolved photographs using the Fourier Slice technique. We also show how several existing super-resolution methods can be seen as particular cases of this approach. Experimental results are provided to show the validity of the approach and its extension to super-resolved all-in-focus image computation and 3D processing is studied.

[1]  Fernando Pérez,et al.  Fourier Slice Super-resolution in plenoptic cameras , 2012, 2012 IEEE International Conference on Computational Photography (ICCP).

[2]  J. H. van Hateren,et al.  Modelling the Power Spectra of Natural Images: Statistics and Information , 1996, Vision Research.

[3]  Marc Levoy,et al.  High performance imaging using large camera arrays , 2005, ACM Trans. Graph..

[4]  Leslie D. Stroebel Photographic materials and processes , 1986 .

[5]  Andrew Lumsdaine,et al.  The focused plenoptic camera , 2009, 2009 IEEE International Conference on Computational Photography (ICCP).

[6]  Edward H. Adelson,et al.  Single Lens Stereo with a Plenoptic Camera , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  F. Okano,et al.  Three-dimensional video system based on integral photography , 1999 .

[8]  H Harashima,et al.  3-D computer graphics based on integral photography. , 2001, Optics express.

[9]  E. Adelson,et al.  The Plenoptic Function and the Elements of Early Vision , 1991 .

[10]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[11]  José Gil Marichal-Hernández,et al.  The Discrete Focal Stack Transform , 2008, 2008 16th European Signal Processing Conference.

[12]  Ren Ng Fourier slice photography , 2005, ACM Trans. Graph..

[13]  Michael Elad,et al.  A fast super-resolution reconstruction algorithm for pure translational motion and common space-invariant blur , 2001, IEEE Trans. Image Process..

[14]  Allen C. Pipkin Fourier and Laplace Transforms , 1986 .

[15]  David Salesin,et al.  Interactive digital photomontage , 2004, ACM Trans. Graph..

[16]  Jrg Kaiser,et al.  Nonrecursive digital filter design using the I-sinh window function , 1977 .

[17]  Robert L. Stevenson,et al.  Super-resolution from image sequences-a review , 1998, 1998 Midwest Symposium on Circuits and Systems (Cat. No. 98CB36268).

[18]  P. Hansen Discrete Inverse Problems: Insight and Algorithms , 2010 .

[19]  Andrew Lumsdaine,et al.  Superresolution with the focused plenoptic camera , 2011, Electronic Imaging.

[20]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[21]  Stephen A. Dyer,et al.  Digital signal processing , 2018, 8th International Multitopic Conference, 2004. Proceedings of INMIC 2004..

[22]  M. Vorontsov,et al.  The principles of adaptive optics , 1985 .

[23]  Andrew Lumsdaine,et al.  Reducing Plenoptic Camera Artifacts , 2010, Comput. Graph. Forum.

[24]  Michael Elad,et al.  Advances and challenges in super‐resolution , 2004, Int. J. Imaging Syst. Technol..

[25]  J. P. Luke,et al.  Simultaneous estimation of super-resolved depth and all-in-focus images from a plenoptic camera , 2009, 2009 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[26]  G. Lippmann Epreuves reversibles donnant la sensation du relief , 1908 .

[27]  Natasha Gelfand,et al.  Generalized autofocus , 2011, 2011 IEEE Workshop on Applications of Computer Vision (WACV).

[28]  Stefan B. Williams,et al.  Decoding, Calibration and Rectification for Lenselet-Based Plenoptic Cameras , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[29]  Shree K. Nayar,et al.  Shape from Focus , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[30]  Tom E. Bishop,et al.  Light field superresolution , 2009, 2009 IEEE International Conference on Computational Photography (ICCP).

[31]  J. V. van Hateren,et al.  Modelling the power spectra of natural images: statistics and information. , 1996, Vision research.

[32]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[33]  David H. Bailey,et al.  The Fractional Fourier Transform and Applications , 1991, SIAM Rev..

[34]  P. Hanrahan,et al.  Digital light field photography , 2006 .

[35]  Moon Gi Kang,et al.  Super-resolution image reconstruction: a technical overview , 2003, IEEE Signal Process. Mag..