Depth from a Light Field Image with Learning-Based Matching Costs

One of the core applications of light field imaging is depth estimation. To acquire a depth map, existing approaches apply a single photo-consistency measure to an entire light field. However, this is not an optimal choice because of the non-uniform light field degradations produced by limitations in the hardware design. In this paper, we introduce a pipeline that automatically determines the best configuration for photo-consistency measure, which leads to the most reliable depth label from the light field. We analyzed the practical factors affecting degradation in lenslet light field cameras, and designed a learning based framework that can retrieve the best cost measure and optimal depth label. To enhance the reliability of our method, we augmented an existing light field benchmark to simulate realistic source dependent noise, aberrations, and vignetting artifacts. The augmented dataset was used for the training and validation of the proposed approach. Our method was competitive with several state-of-the-art methods for the benchmark and real-world light field datasets.

[1]  Stefan B. Williams,et al.  Decoding, Calibration and Rectification for Lenselet-Based Plenoptic Cameras , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Andreas Klaus,et al.  Segment-Based Stereo Matching Using Belief Propagation and a Self-Adapting Dissimilarity Measure , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[3]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[4]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[5]  Shree K. Nayar,et al.  PiCam , 2013, ACM Trans. Graph..

[6]  Sven Wanner,et al.  Datasets and Benchmarks for Densely Sampled 4D Light Fields , 2013, VMV.

[7]  Stefan B. Williams,et al.  Light field image denoising using a linear 4D frequency-hyperfan all-in-focus filter , 2013, Electronic Imaging.

[8]  Ramin Zabih,et al.  Non-parametric Local Transforms for Computing Visual Correspondence , 1994, ECCV.

[9]  Chao Li,et al.  Robust depth estimation for light field via spinning parallelogram operator , 2016, Comput. Vis. Image Underst..

[10]  Alexei A. Efros,et al.  Depth Estimation with Occlusion Modeling Using Light-Field Cameras , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Bastian Goldlücke,et al.  What Sparse Light Field Coding Reveals about Scene Structure , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Alexei A. Efros,et al.  SVBRDF-Invariant Shape and Reflectance Estimation from Light-Field Cameras , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Raquel Urtasun,et al.  Efficient Joint Segmentation, Occlusion Labeling, Stereo and Flow Estimation , 2014, ECCV.

[14]  Bastian Goldlücke,et al.  A Dataset and Evaluation Methodology for Depth Estimation on 4D Light Fields , 2016, ACCV.

[15]  Jitendra Malik,et al.  Depth Estimation and Specular Removal for Glossy Surfaces Using Point and Line Consistency with Light-Field Cameras , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Williem,et al.  Robust Light Field Depth Estimation for Noisy Scene with Occlusion , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Thomas Pock,et al.  Shape from Light Field Meets Robust PCA , 2014, ECCV.

[18]  P. Hanrahan,et al.  Light Field Photography with a Hand-held Plenoptic Camera , 2005 .

[19]  Raquel Urtasun,et al.  Efficient Deep Learning for Stereo Matching , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Zhan Yu,et al.  Line Assisted Light Field Triangulation and Stereo Matching , 2013, 2013 IEEE International Conference on Computer Vision.

[21]  Marc Levoy,et al.  High performance imaging using large camera arrays , 2005, ACM Trans. Graph..

[22]  Youngbae Hwang,et al.  Difference-Based Image Noise Modeling Using Skellam Distribution , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Kuk-Jin Yoon,et al.  Leveraging stereo matching with learning-based confidence measures , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Zhan Yu,et al.  Light Field Stereo Matching Using Bilateral Statistics of Surface Cameras , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  Jitendra Malik,et al.  Depth from Combining Defocus and Correspondence Using Light-Field Cameras , 2013, 2013 IEEE International Conference on Computer Vision.

[26]  Shree K. Nayar,et al.  Multiplexing for Optimal Lighting , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  In-So Kweon,et al.  Accurate depth map estimation from a lenslet light field camera , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  In-So Kweon,et al.  Geometric Calibration of Micro-Lens-Based Light Field Cameras Using Line Features , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Carsten Rother,et al.  Fast cost-volume filtering for visual correspondence and beyond , 2011, CVPR 2011.

[30]  Yael Pritch,et al.  Scene reconstruction from high spatio-angular resolution light fields , 2013, ACM Trans. Graph..

[31]  Xing Mei,et al.  On building an accurate stereo matching system on graphics hardware , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[32]  C.E. Shannon,et al.  Communication in the Presence of Noise , 1949, Proceedings of the IRE.

[33]  Pat Hanrahan,et al.  Digital correction of lens aberrations in light field photography , 2006, International Optical Design Conference.

[34]  Tom E. Bishop,et al.  The Light Field Camera: Extended Depth of Field, Aliasing, and Superresolution , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Sven Wanner,et al.  Variational Light Field Analysis for Disparity Estimation and Super-Resolution , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Bastian Goldlücke,et al.  On the Calibration of Focused Plenoptic Cameras , 2013, Time-of-Flight and Depth Imaging.

[37]  Jonathan M. Garibaldi,et al.  Real-Time Correlation-Based Stereo Vision with Reduced Border Errors , 2002, International Journal of Computer Vision.

[38]  Juho Kannala,et al.  A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[39]  Stefano Mattoccia,et al.  Deep Stereo Fusion: Combining Multiple Disparity Hypotheses with Deep-Learning , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[40]  Yann LeCun,et al.  Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches , 2015, J. Mach. Learn. Res..

[41]  Sven Wanner,et al.  The Variational Structure of Disparity and Regularization of 4D Light Fields , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[42]  Enhua Wu,et al.  Constant Time Weighted Median Filtering for Stereo Matching and Beyond , 2013, 2013 IEEE International Conference on Computer Vision.

[43]  Heiko Hirschmüller,et al.  Evaluation of Stereo Matching Costs on Images with Radiometric Differences , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[44]  Kiriakos N. Kutulakos,et al.  What does an aberrated photo tell us about the lens and the scene? , 2013, IEEE International Conference on Computational Photography (ICCP).

[45]  Ravi Ramamoorthi,et al.  A Light Transport Framework for Lenslet Light Field Cameras , 2015, TOGS.

[46]  Jitendra Malik,et al.  Shape Estimation from Shading, Defocus, and Correspondence Using Light-Field Angular Coherence , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Sergio Orts,et al.  HyperDepth: Learning Depth from Structured Light without Matching , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Philippos Mordohai,et al.  Ensemble Classifier for Combining Stereo Matching Algorithms , 2015, 2015 International Conference on 3D Vision.

[49]  Andrew Lumsdaine,et al.  Reducing Plenoptic Camera Artifacts , 2010, Comput. Graph. Forum.

[50]  Ashok Veeraraghavan,et al.  Light field denoising, light field superresolution and stereo camera based refocussing using a GMM light field patch prior , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.