Focal Sweep Camera for Space-Time Refocusing

In this paper, we present an imaging system that 1 captures a sequence of images in a scene as focus is swept 2 over a large depth range. We refer to it as a space-time focal 3 stack and propose algorithms for space-time image refocus4 ing at sensor resolution – when a user clicks on a pixel, we 5 show the user the image that was captured at the time when 6 the pixel is best focused. 7 We first describe how to synchronize focus sweeping 8 with image capturing so that the summed depth of field of 9 a focal stack efficiently covers the entire depth range with10 out gaps or overlap. We then propose an efficient algorithm 11 for computing a space-time in-focus index map from a fo12 cal stack. It states which time each pixel is best focused 13 and the algorithm is specifically designed to enable a seam14 less refocusing experience, even in textureless regions or at 15 depth discontinuities. We implemented two prototype cam16 eras to capture a large number of space-time focal stacks, 17 and made an interactive refocusing viewer available online 18 at www.focalsweep.com. 19

[1]  Frédo Durand,et al.  Image and depth from a conventional camera with a coded aperture , 2007, SIGGRAPH 2007.

[2]  Jitendra Malik,et al.  Scale-Space and Edge Detection Using Anisotropic Diffusion , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Subhasis Chaudhuri,et al.  Optimal selection of camera parameters for recovery of depth from defocused images , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[4]  J. Conchello,et al.  Three-dimensional imaging by deconvolution microscopy. , 1999, Methods.

[5]  David Salesin,et al.  Interactive digital photomontage , 2004, SIGGRAPH 2004.

[6]  F. Meriaudeau,et al.  Depth from Defocus , 2021, Computer Vision.

[7]  Shree K. Nayar,et al.  Shape from focus: an effective approach for rough surfaces , 1990, Proceedings., IEEE International Conference on Robotics and Automation.

[8]  J. Sibarita.  Deconvolution microscopy. , 2005, Advances in biochemical engineering/biotechnology.

[9]  P. Hanrahan,et al.  Light Field Photography with a Hand-held Plenoptic Camera , 2005 .

[10]  Leonard McMillan,et al.  Dynamically reparameterized light fields , 2000, SIGGRAPH.

[11]  Marc Levoy,et al.  High performance imaging using large camera arrays , 2005, SIGGRAPH 2005.

[12]  Oliver Cossairt,et al.  Spectral Focal Sweep: Extended depth of field from chromatic aberrations , 2010, 2010 IEEE International Conference on Computational Photography (ICCP).

[13]  W. Freeman,et al.  Generalized Belief Propagation , 2000, NIPS.

[14]  Jorge Ojeda-Castañeda,et al.  Asymmetric phase masks for extended depth of field. , 2004, Applied optics.

[15]  Ramesh Raskar,et al.  Glare aware photography: 4D ray sampling for reducing glare effects of camera lenses , 2008, ACM Trans. Graph..

[16]  Hai Tao,et al.  A global matching framework for stereo computation , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[17]  Kiriakos N. Kutulakos,et al.  Time-constrained photography , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[18]  M Motamedi,et al.  Optical/digital incoherent image processing for extended depth of field. , 1987, Applied optics.

[19]  G. Häusler,et al.  A method to increase the depth of focus by two step image processing , 1972 .

[20]  Shin-Tson Wu,et al.  Variable-focus liquid lens. , 2007, Optics express.

[21]  Muralidhara Subbarao,et al.  Selecting the Optimal Focus Measure for Autofocusing and Depth-From-Focus , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[22]  Wojciech Matusik,et al.  3D TV: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes , 2004, ACM Trans. Graph..

[23]  Pantazis Mouroulis.  Depth of field extension with spherical optics. , 2008, Optics express.

[24]  Bahram Javidi,et al.  Three-Dimensional Television, Video and Display Technology , 2002 .

[25]  Shree K. Nayar,et al.  Real-Time Focus Range Sensor , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Vladimir Kolmogorov,et al.  An experimental comparison of min-cut/max- flow algorithms for energy minimization in vision , 2001, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[28]  Herbert E. Ives,et al.  Parallax Panoramagrams Made with a Large Diameter Lens , 1930 .

[29]  Nicholas George,et al.  Extended depth of field using a logarithmic asphere , 2003 .

[30]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[31]  G Indebetouw,et al.  Imaging with Fresnel zone pupil masks: extended depth of field. , 1984, Applied optics.

[32]  Frédéric Guichard,et al.  Extended depth-of-field using sharpness transport across color channels , 2009, Electronic Imaging.

[33]  David Salesin,et al.  Spatio-angular resolution tradeoffs in integral photography , 2006, EGSR '06.

[34]  M. Levoy,et al.  Light field microscopy , 2006, SIGGRAPH 2006.

[35]  Shree K. Nayar,et al.  Transactions on Pattern Analysis and Machine Intelligence Flexible Depth of Field Photography 1 Depth of Field , 2022 .

[36]  Subhasis Chaudhuri,et al.  Depth From Defocus: A Real Aperture Imaging Approach , 1999, Springer New York.

[37]  S. Nayar,et al.  Diffusion coded photography for extended depth of field , 2010, SIGGRAPH 2010.

[38]  Shree K. Nayar,et al.  Shape from Focus , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[39]  Frédo Durand,et al.  Image and depth from a conventional camera with a coded aperture , 2007, ACM Trans. Graph..

[40]  Kiriakos N. Kutulakos.  Focal Stack Photography: High-Performance Photography with a Conventional Camera , 2009, MVA.

[41]  Stephen Lin,et al.  Coded aperture pairs for depth from defocus , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[42]  Chintan Intwala,et al.  Light Field Camera Design for Integral View Photography , 2006 .

[43]  W. Cathey,et al.  Extended depth of field through wave-front coding. , 1995, Applied optics.

[44]  Shin‐Tson Wu,et al.  Tunable-focus liquid lens controlled using a servo motor. , 2006, Optics express.

[45]  P. Hanrahan,et al.  Digital light field photography , 2006 .

[46]  Kiriakos N. Kutulakos,et al.  Light-Efficient Photography , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Alex Pentland,et al.  A New Sense for Depth of Field , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[48]  Steven A. Shafer,et al.  Depth from focusing and defocusing , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[49]  SubbaraoMurali,et al.  Depth from defocus , 1994 .