Depth Recovery from Light Field Using Focal Stack Symmetry

We describe a technique to recover depth from a light field (LF) using two proposed features of the LF focal stack. One feature is the property that non-occluding pixels exhibit symmetry along the focal depth dimension centered at the in-focus slice. The other is a data consistency measure based on analysis-by-synthesis, i.e., the difference between the synthesized focal stack given the hypothesized depth map and that from the LF. These terms are used in an iterative optimization framework to extract scene depth. Experimental results on real Lytro and Raytrix data demonstrate that our technique outperforms state-of-the-art solutions and is significantly more robust to noise and under-sampling.

[1]  Wojciech Matusik,et al.  Multi-perspective stereoscopy from light fields , 2011, ACM Trans. Graph..

[2]  Sven Wanner,et al.  Globally Consistent Multi-label Assignment on the Ray Space of 4D Light Fields , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[3]  Pierre Vandergheynst,et al.  A Convex Solution to Disparity Estimation from Light Fields via the Primal-Dual Method , 2015, EMMCVPR.

[4]  Subhasis Chaudhuri,et al.  Depth estimation and image restoration using defocused stereo pairs , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Andrew Lumsdaine,et al.  Plenoptic rendering with interactive performance using GPUs , 2012, Electronic Imaging.

[6]  Shree K. Nayar,et al.  Depth from Diffusion , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[7]  Yael Pritch,et al.  Scene reconstruction from high spatio-angular resolution light fields , 2013, ACM Trans. Graph..

[8]  Francesc Moreno-Noguer,et al.  Active refocusing of images and videos , 2007, ACM Trans. Graph..

[9]  Sven Wanner,et al.  Datasets and Benchmarks for Densely Sampled 4D Light Fields , 2013, VMV.

[10]  Jürgen Beyerer,et al.  A New Approach for Estimating Depth by Fusing Stereo and Defocus Information , 2007, GI Jahrestagung.

[11]  Shree K. Nayar,et al.  Real-time focus range sensor , 1995, Proceedings of IEEE International Conference on Computer Vision.

[12]  Stefan B. Williams,et al.  Decoding, Calibration and Rectification for Lenselet-Based Plenoptic Cameras , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[13]  Michael Heizmann,et al.  Fusion of Combined Stereo and Focus Series for Depth Estimation , 2006, GI Jahrestagung.

[14]  Stefano Soatto,et al.  This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE IEEE TRANSACTION OF PATTERN RECO , 2022 .

[15]  Sven Wanner,et al.  Variational Light Field Analysis for Disparity Estimation and Super-Resolution , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Yoav Y. Schechner,et al.  Depth from Defocus vs. Stereo: How Different Really Are They? , 2004, International Journal of Computer Vision.

[17]  Marc Levoy,et al.  Reconstructing Occluded Surfaces Using Synthetic Apertures: Stereo, Focus and Robust Measures , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[18]  Tae-Sun Choi,et al.  Depth Map Estimation using a Robust Focus Measure , 2007, 2007 IEEE International Conference on Image Processing.

[19]  Haibin Ling,et al.  Saliency Detection on Light Field , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Thomas Pock,et al.  Variational Shape from Light Field , 2013, EMMCVPR.

[21]  Sven Wanner,et al.  Spatial and Angular Variational Super-Resolution of 4D Light Fields , 2012, ECCV.

[22]  Feng Li,et al.  Dual-focus stereo imaging , 2010, J. Electronic Imaging.

[23]  Shree K. Nayar,et al.  Multiple view image denoising , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Christian Frese,et al.  Robust Depth Estimation by Fusion of Stereo and Focus Series Acquired with a Camera Array , 2006, 2006 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems.

[25]  Stefano Soatto,et al.  A geometric approach to shape from defocus , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Alessandro Foi,et al.  Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering , 2007, IEEE Transactions on Image Processing.

[27]  Shree K. Nayar,et al.  Shape from focus system , 1992, Proceedings 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[28]  Ramon M. Rodriguez-Dagnino,et al.  Synthesizing stereo 3D views from focus cues in monoscopic 2D images , 2003, IS&T/SPIE Electronic Imaging.

[29]  Horst Bischof,et al.  Image Guided Depth Upsampling Using Anisotropic Total Generalized Variation , 2013, 2013 IEEE International Conference on Computer Vision.

[30]  Shree K. Nayar,et al.  Shape from Focus , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[31]  Zhan Yu,et al.  Light Field Stereo Matching Using Bilateral Statistics of Surface Cameras , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[32]  Jitendra Malik,et al.  Depth from Combining Defocus and Correspondence Using Light-Field Cameras , 2013, 2013 IEEE International Conference on Computer Vision.

[33]  Thomas Pock,et al.  Shape from Light Field Meets Robust PCA , 2014, ECCV.

[34]  Michael J. Black,et al.  Secrets of optical flow estimation and their principles , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[35]  P. Hanrahan,et al.  Light Field Photography with a Hand-held Plenoptic Camera , 2005 .

[36]  Zhan Yu,et al.  Line Assisted Light Field Triangulation and Stereo Matching , 2013, 2013 IEEE International Conference on Computer Vision.

[37]  Chih-Hung Liu,et al.  Depth map generation based on depth from focus , 2010, 2010 International Conference on Electronic Devices, Systems and Applications.