Depth Estimation Through a Generative Model of Light Field Synthesis

Light field photography captures rich structural information that may facilitate a number of traditional image processing and computer vision tasks. A crucial ingredient in such endeavors is accurate depth recovery. We present a novel framework that allows the recovery of a high quality continuous depth map from light field data. To this end we propose a generative model of a light field that is fully parametrized by its corresponding depth map. The model allows for the integration of powerful regularization techniques such as a non-local means prior, facilitating accurate depth map estimation. Comparisons with previous methods show that we are able to recover faithful depth maps with much finer details. In a number of challenging real-world examples we demonstrate both the effectiveness and robustness of our approach.

[1]  Alexei A. Efros,et al.  Occlusion-Aware Depth Estimation Using Light-Field Cameras , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[2]  Sven Wanner,et al.  Globally Consistent Multi-label Assignment on the Ray Space of 4D Light Fields , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[3]  Sven Wanner,et al.  Datasets and Benchmarks for Densely Sampled 4D Light Fields , 2013, VMV.

[4]  Can Chen,et al.  Depth Recovery from Light Field Using Focal Stack Symmetry , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[5]  Jean-Michel Morel,et al.  A non-local algorithm for image denoising , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[6]  Lennart Wietzke,et al.  Single lens 3D-camera with extended depth-of-field , 2012, Electronic Imaging.

[7]  Sven Wanner,et al.  The Variational Structure of Disparity and Regularization of 4D Light Fields , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Yael Pritch,et al.  Scene reconstruction from high spatio-angular resolution light fields , 2013, ACM Trans. Graph..

[9]  Thomas Pock,et al.  Shape from Light Field Meets Robust PCA , 2014, ECCV.

[10]  Paolo Favaro,et al.  Recovering thin structures via nonlocal-means regularization with application to depth from defocus , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[11]  Ivana Tosic,et al.  Light Field Scale-Depth Space Transform for Dense Depth Estimation , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[12]  Stefan B. Williams,et al.  Light field image denoising using a linear 4D frequency-hyperfan all-in-focus filter , 2013, Electronic Imaging.

[13]  P. Hanrahan,et al.  Digital light field photography , 2006 .

[14]  Stefan B. Williams,et al.  Plenoptic flow: Closed-form visual odometry for light field cameras , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[15]  Sven Wanner,et al.  Globally consistent depth labeling of 4D light fields , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Edward H. Adelson,et al.  Single Lens Stereo with a Plenoptic Camera , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  Stefan B. Williams,et al.  Decoding, Calibration and Rectification for Lenselet-Based Plenoptic Cameras , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Bernd Girod,et al.  Multi-view Geometry Estimation for Light Field Compression , 2002, VMV.

[19]  Harry Shum,et al.  Plenoptic sampling , 2000, SIGGRAPH.

[20]  Haibin Ling,et al.  Saliency Detection on Light Field , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Leonard McMillan,et al.  Dynamically reparameterized light fields , 2000, SIGGRAPH.

[22]  Chia-Kai Liang,et al.  Programmable aperture photography: multiplexed light field acquisition , 2008, SIGGRAPH 2008.

[23]  Yu-Wing Tai,et al.  Consistent Matting for Light Field Images , 2014, ECCV.

[24]  Horst Bischof,et al.  Image Guided Depth Upsampling Using Anisotropic Total Generalized Variation , 2013, 2013 IEEE International Conference on Computer Vision.

[25]  Jitendra Malik,et al.  Depth from Combining Defocus and Correspondence Using Light-Field Cameras , 2013, 2013 IEEE International Conference on Computer Vision.

[26]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[27]  Tom E. Bishop,et al.  The Light Field Camera: Extended Depth of Field, Aliasing, and Superresolution , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Sven Wanner,et al.  Variational Light Field Analysis for Disparity Estimation and Super-Resolution , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Marc Levoy,et al.  Using plane + parallax for calibrating dense camera arrays , 2004, CVPR 2004.

[30]  Michael S. Brown,et al.  High quality depth map upsampling for 3D-TOF cameras , 2011, 2011 International Conference on Computer Vision.

[31]  Bastian Goldlücke,et al.  Epipolar Plane Image Refocusing for Improved Depth Estimation and Occlusion Handling , 2013, VMV.

[32]  Michael J. Black,et al.  Secrets of optical flow estimation and their principles , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[33]  Thomas Pock,et al.  Variational Shape from Light Field , 2013, EMMCVPR.

[34]  Sven Wanner,et al.  Spatial and Angular Variational Super-Resolution of 4D Light Fields , 2012, ECCV.

[35]  Qionghai Dai,et al.  Light field from micro-baseline image pair , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[37]  Marc Levoy,et al.  Using plane + parallax for calibrating dense camera arrays , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[38]  Jorge Nocedal,et al.  Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization , 1997, TOMS.

[39]  P. Hanrahan,et al.  Light Field Photography with a Hand-held Plenoptic Camera , 2005 .

[40]  Robert C. Bolles,et al.  Epipolar-plane image analysis: An approach to determining structure from motion , 1987, International Journal of Computer Vision.