Depth Estimation From Light Field Using Graph-Based Structure-Aware Analysis

Existing light field depth map estimation approaches only utilize partial angular views in occlusion areas and local spatial dependencies in the optimization. This paper proposes a novel two-stage light field depth estimation method via graph spectral analysis to exploit the complete correlations and dependencies within angular patches and spatial images. The initial depth map estimation leverages the undirected graph to jointly consider occluded and unoccluded views within each angular patch. The estimated depth minimizes the structural incoherence of its corresponding angular patch with the focused one by evaluating the highest graph frequency component. Subsequently, depth map refinement optimizes the initial depth map with the color consistency and smoothness formulated by weighted adjacency matrix. The structural constraints are efficiently employed using low-pass graph filtering with Chebyshev polynomial approximation. Experimental results demonstrate that the proposed method improves the depth map estimation, especially in the edge regions.

[1]  Hiroshi Ishikawa,et al.  Exact Optimization for Markov Random Fields with Convex Priors , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  P. Hanrahan,et al.  Light Field Photography with a Hand-held Plenoptic Camera , 2005 .

[3]  Andrew Lumsdaine,et al.  Reducing Plenoptic Camera Artifacts , 2010, Comput. Graph. Forum.

[4]  Daniel Cremers,et al.  Global Solutions of Variational Models with Convex Regularization , 2010, SIAM J. Imaging Sci..

[5]  Sunil K. Narang,et al.  Downsampling graphs using spectral theory , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[6]  Tom E. Bishop,et al.  The Light Field Camera: Extended Depth of Field, Aliasing, and Superresolution , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Sven Wanner,et al.  Globally consistent depth labeling of 4D light fields , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Sunil K. Narang,et al.  Compact Support Biorthogonal Wavelet Filterbanks for Arbitrary Undirected Graphs , 2012, IEEE Transactions on Signal Processing.

[9]  Zhan Yu,et al.  Line Assisted Light Field Triangulation and Stereo Matching , 2013, 2013 IEEE International Conference on Computer Vision.

[10]  Pascal Frossard,et al.  The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains , 2012, IEEE Signal Processing Magazine.

[11]  Yael Pritch,et al.  Scene reconstruction from high spatio-angular resolution light fields , 2013, ACM Trans. Graph..

[12]  José M. F. Moura,et al.  Discrete Signal Processing on Graphs , 2012, IEEE Transactions on Signal Processing.

[13]  Sunil K. Narang,et al.  Bilateral filter: Graph spectral interpretation and extensions , 2013, 2013 IEEE International Conference on Image Processing.

[14]  Jitendra Malik,et al.  Depth from Combining Defocus and Correspondence Using Light-Field Cameras , 2013, 2013 IEEE International Conference on Computer Vision.

[15]  José M. F. Moura,et al.  Discrete Signal Processing on Graphs: Frequency Analysis , 2013, IEEE Transactions on Signal Processing.

[16]  Zhan Yu,et al.  Light Field Stereo Matching Using Bilateral Statistics of Surface Cameras , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Thomas Pock,et al.  Shape from Light Field Meets Robust PCA , 2014, ECCV.

[18]  José M. F. Moura,et al.  Signal denoising on graphs via graph filtering , 2014, 2014 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[19]  Sven Wanner,et al.  Variational Light Field Analysis for Disparity Estimation and Super-Resolution , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  In-So Kweon,et al.  Accurate depth map estimation from a lenslet light field camera , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Antonio Ortega,et al.  Intra-Prediction and Generalized Graph Fourier Transform for Image Coding , 2015, IEEE Signal Processing Letters.

[22]  Oscar C. Au,et al.  Multiresolution Graph Fourier Transform for Compression of Piecewise Smooth Images , 2015, IEEE Transactions on Image Processing.

[23]  Mingliang Chen,et al.  A real-time virtual dressing system with RGB-D camera , 2015, 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA).

[24]  Bastian Goldlücke,et al.  What Sparse Light Field Coding Reveals about Scene Structure , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Yuchen Zhang,et al.  A cost minimization with light field in scene depth MAP estimation , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[26]  Alexei A. Efros,et al.  Depth Estimation with Occlusion Modeling Using Light-Field Cameras , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Thomas Pock,et al.  Convolutional Networks for Shape from Light Field , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Jitendra Malik,et al.  Depth Estimation and Specular Removal for Glossy Surfaces Using Point and Line Consistency with Light-Field Cameras , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Gene Cheung,et al.  Graph Laplacian Regularization for Image Denoising: Analysis in the Continuous Domain , 2016, IEEE Transactions on Image Processing.

[30]  Jie Chen,et al.  Light Field Compressed Sensing Over a Disparity-Aware Dictionary , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[31]  Qing Wang,et al.  Occlusion-Model Guided Antiocclusion Depth Estimation in Light Field , 2016, IEEE Journal of Selected Topics in Signal Processing.

[32]  Gerald Matz,et al.  Graph Signal Recovery via Primal-Dual Algorithms for Total Variation Minimization , 2017, IEEE Journal of Selected Topics in Signal Processing.

[33]  Pascal Frossard,et al.  Graph-based light field super-resolution , 2017, 2017 IEEE 19th International Workshop on Multimedia Signal Processing (MMSP).

[34]  Qian Huang,et al.  Light-Field Depth Estimation via Epipolar Plane Image Analysis and Locally Linear Embedding , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[35]  Pascal Frossard,et al.  Geometry-Consistent Light Field Super-Resolution via Graph-Based Regularization , 2017, IEEE Transactions on Image Processing.

[36]  Hermina Petric Maretic,et al.  A graph learning approach for light field image compression , 2018, Optical Engineering + Applications.

[37]  Yaonan Wang,et al.  Benchmark Data Set and Method for Depth Estimation From Light Field Images , 2018, IEEE Transactions on Image Processing.

[38]  Pascal Frossard,et al.  A Nonsmooth Graph-Based Approach to Light Field Super-Resolution , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[39]  Wu Luo,et al.  Key Joints Selection and Spatiotemporal Mining for Skeleton-Based Action Recognition , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[40]  Jie Chen,et al.  Light Field Image Compression Based on Bi-Level View Compensation With Rate-Distortion Optimization , 2019, IEEE Transactions on Circuits and Systems for Video Technology.

[41]  Xilin Chen,et al.  Hyperspectral Light Field Stereo Matching , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  In So Kweon,et al.  Depth from a Light Field Image with Learning-Based Matching Costs , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Ping Li,et al.  Deep Color Guided Coarse-to-Fine Convolutional Network Cascade for Depth Image Super-Resolution , 2019, IEEE Transactions on Image Processing.

[44]  Chao-Tsung Huang,et al.  Empirical Bayesian Light-Field Stereo Matching by Robust Pseudo Random Field Modeling , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[45]  Junsong Yuan,et al.  Dictionary Learning-Based, Directional, and Optimized Prediction for Lenslet Image Coding , 2019, IEEE Transactions on Circuits and Systems for Video Technology.