Scale and Orientation Aware EPI-Patch Learning for Light Field Depth Estimation

Epipolar Plane Image (EPI) implies some important depth cues for light field depth estimation. Intuitively, the EPI patches with different spatial scales and orientations may exhibit different features and result in different estimation precision. In this paper, we discuss this issue and present a scale and orientation aware EPI-Patch learning model for depth estimation. We take the multi-orientation EPI patches of each pixel as input, and design two types of network structures for adaptive scale selection and orientation fusion. One type is a scale-aware structure, which feeds one orientation patch into a multi-layer feed-forward network with long and short skip connections. The other type is a shared-weight network for fusing the multi-orientation features. We demonstrate the effectiveness of our model by experiments on 4D Light Field Benchmark.

[1]  Qian Huang,et al.  Light-Field Depth Estimation via Epipolar Plane Image Analysis and Locally Linear Embedding , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[2]  Haibin Ling,et al.  Saliency Detection on Light Field , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Yael Pritch,et al.  Scene reconstruction from high spatio-angular resolution light fields , 2013, ACM Trans. Graph..

[4]  Sven Wanner,et al.  Datasets and Benchmarks for Densely Sampled 4D Light Fields , 2013, VMV.

[5]  Guojun Dai,et al.  EPI-Patch Based Convolutional Neural Network for Depth Estimation on 4D Light Field , 2017, ICONIP.

[6]  In-So Kweon,et al.  A Taxonomy and Evaluation of Dense Light Field Depth Estimation Algorithms , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[7]  Camille Couprie,et al.  Learning Hierarchical Features for Scene Labeling , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  D. Yang,et al.  Occlusion-aware depth estimation for light field using multi-orientation EPIs , 2018, Pattern Recognit..

[9]  Alessandro Neri,et al.  A multi-resolution approach to depth field estimation in dense image arrays , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[10]  Jitendra Malik,et al.  Depth from Combining Defocus and Correspondence Using Light-Field Cameras , 2013, 2013 IEEE International Conference on Computer Vision.

[11]  Chao Li,et al.  Robust depth estimation for light field via spinning parallelogram operator , 2016, Comput. Vis. Image Underst..

[12]  Alexei A. Efros,et al.  Depth Estimation with Occlusion Modeling Using Light-Field Cameras , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Alex Kendall,et al.  End-to-End Learning of Geometry and Context for Deep Stereo Regression , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[14]  Bastian Goldlücke,et al.  A Dataset and Evaluation Methodology for Depth Estimation on 4D Light Fields , 2016, ACCV.

[15]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Andrew Lumsdaine,et al.  The focused plenoptic camera , 2009, 2009 IEEE International Conference on Computational Photography (ICCP).

[17]  Bastian Goldlücke,et al.  What Sparse Light Field Coding Reveals about Scene Structure , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Yi Yang,et al.  Attention to Scale: Scale-Aware Semantic Image Segmentation , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Can Chen,et al.  Depth Recovery from Light Field Using Focal Stack Symmetry , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[20]  In-So Kweon,et al.  Accurate depth map estimation from a lenslet light field camera , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Christopher Joseph Pal,et al.  The Importance of Skip Connections in Biomedical Image Segmentation , 2016, LABELS/DLMIA@MICCAI.

[22]  In So Kweon,et al.  Depth from a Light Field Image with Learning-Based Matching Costs , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Gordon Wetzstein,et al.  Compressive light field photography using overcomplete dictionaries and optimized projections , 2013, ACM Trans. Graph..

[24]  P. Hanrahan,et al.  Light Field Photography with a Hand-held Plenoptic Camera , 2005 .

[25]  Robert C. Bolles,et al.  Epipolar-plane image analysis: An approach to determining structure from motion , 1987, International Journal of Computer Vision.

[26]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Jitendra Malik,et al.  Shape Estimation from Shading, Defocus, and Correspondence Using Light-Field Angular Coherence , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Sven Wanner,et al.  Globally consistent depth labeling of 4D light fields , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[29]  Touradj Ebrahimi,et al.  New Light Field Image Dataset , 2016, QoMEX 2016.