VommaNet: an End-to-End Network for Disparity Estimation from Reflective and Texture-less Light Field Images

The precise combination of image sensor and micro-lens array enables lenslet light field cameras to record both angular and spatial information of incoming light, therefore, one can calculate disparity and depth from light field images. In turn, 3D models of the recorded objects can be recovered, which is a great advantage over other imaging system. However, reflective and texture-less areas in light field images have complicated conditions, making it hard to correctly calculate disparity with existing algorithms. To tackle this problem, we introduce a novel end-to-end network VommaNet to retrieve multi-scale features from reflective and texture-less regions for accurate disparity estimation. Meanwhile, our network has achieved similar or better performance in other regions for both synthetic light field images and real-world data compared to the state-of-the-art algorithms. Currently, we achieve the best score for mean squared error (MSE) on HCI 4D Light Field Benchmark.

[1]  In So Kweon,et al.  Geometric Calibration of Micro-Lens-Based Light-Field Cameras Using Line Features , 2014, ECCV.

[2]  Thomas Pock,et al.  Convolutional Networks for Shape from Light Field , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Silvio Savarese,et al.  3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction , 2016, ECCV.

[4]  François Chollet,et al.  Xception: Deep Learning with Depthwise Separable Convolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[6]  In-So Kweon,et al.  Accurate depth map estimation from a lenslet light field camera , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Ko Nishino,et al.  Multiview Shape and Reflectance from Natural Illumination , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  In-So Kweon,et al.  EPINET: A Fully-Convolutional Neural Network Using Epipolar Geometry for Depth from Light Field Images , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[9]  Marc Pollefeys,et al.  Direction matters: Depth estimation with a surface normal classifier , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Jan Kautz,et al.  Polarimetric Multi-view Stereo , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  P. Hanrahan,et al.  Digital light field photography , 2006 .

[12]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Marc Levoy,et al.  Light Fields and Computational Imaging , 2006, Computer.

[14]  Sven Wanner,et al.  Reconstructing Reflective and Transparent Surfaces from Epipolar Plane Images , 2013, GCPR.

[15]  Bastian Goldlücke,et al.  Light Field Intrinsics with a Deep Encoder-Decoder Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[16]  Yaonan Wang,et al.  3D Face Reconstruction from Light Field Images: A Model-free Approach , 2017, ECCV.

[17]  Chao Li,et al.  Robust depth estimation for light field via spinning parallelogram operator , 2016, Comput. Vis. Image Underst..

[18]  Yasuyuki Matsushita,et al.  High-quality shape from multi-view stereo and shading under general illumination , 2011, CVPR 2011.

[19]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[20]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[21]  Jingyi Yu,et al.  Light field imaging: models, calibrations, reconstructions, and applications , 2017, Frontiers of Information Technology & Electronic Engineering.

[22]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Jitendra Malik,et al.  Shape Estimation from Shading, Defocus, and Correspondence Using Light-Field Angular Coherence , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Michael Goesele,et al.  Shading-Aware Multi-view Stereo , 2016, ECCV.

[25]  Sven Wanner,et al.  Variational Light Field Analysis for Disparity Estimation and Super-Resolution , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  In So Kweon,et al.  Depth from a Light Field Image with Learning-Based Matching Costs , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Robert C. Bolles,et al.  Epipolar-plane image analysis: An approach to determining structure from motion , 1987, International Journal of Computer Vision.

[28]  Bastian Goldlücke,et al.  What Sparse Light Field Coding Reveals about Scene Structure , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Takayuki Okatani,et al.  Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps With Accurate Object Boundaries , 2018, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[30]  Bastian Goldlücke,et al.  A Dataset and Evaluation Methodology for Depth Estimation on 4D Light Fields , 2016, ACCV.

[31]  D. Yang,et al.  Occlusion-aware depth estimation for light field using multi-orientation EPIs , 2018, Pattern Recognit..

[32]  Bernd Jähne,et al.  Trust your Model: Light Field Depth Estimation with Inline Occlusion Handling , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[33]  Wei Yu,et al.  Neural EPI-Volume Networks for Shape from Light Field , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[34]  Stephan Hussmann,et al.  Automated Robust Metric Calibration Algorithm for Multifocus Plenoptic Cameras , 2016, IEEE Transactions on Instrumentation and Measurement.