Intrinsic Light Field Decomposition and Disparity Estimation with Deep Encoder-Decoder Network

We present an encoder-decoder deep neural network that solves non-Lambertian intrinsic light field decomposition, where we recover all three intrinsic components: albedo, shading, and specularity. We learn a sparse set of features from 3D epipolar volumes and use them in separate decoder pathways to reconstruct intrinsic light fields. While being trained on synthetic data generated with Blender, our model still generalizes to real world examples captured with a Lytro Illum plenoptic camera. The proposed method outperforms state-of-the-art approaches for single images and achieves competitive accuracy with recent modeling methods for light fields.

[1]  Bastian Goldlücke,et al.  A Variational Model for Intrinsic Light Field Decomposition , 2016, ACCV.

[2]  Bastian Goldlücke,et al.  What Sparse Light Field Coding Reveals about Scene Structure , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Stella X. Yu,et al.  Direct Intrinsics: Learning Albedo-Shading Decomposition by Convolutional Regression , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[4]  Bastian Goldlücke,et al.  Shadow and Specularity Priors for Intrinsic Light Field Decomposition , 2017, EMMCVPR.

[5]  Steven A. Shafer,et al.  Using color to separate reflection components , 1985 .

[6]  Bastian Goldlücke,et al.  Light Field Intrinsics with a Deep Encoder-Decoder Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[7]  Kun Zhou,et al.  Intrinsic Light Field Images , 2016, Comput. Graph. Forum.

[8]  Richard Szeliski,et al.  On the Motion and Appearance of Specularities in Image Sequences , 2002, ECCV.

[9]  Bastian Goldlücke,et al.  A Dataset and Evaluation Methodology for Depth Estimation on 4D Light Fields , 2016, ACCV.

[10]  Sven Wanner,et al.  Globally consistent depth labeling of 4D light fields , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Balazs Kovacs,et al.  Intrinsic Decompositions for Image Editing , 2017, Comput. Graph. Forum.

[12]  Sepp Hochreiter,et al.  Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) , 2015, ICLR.

[13]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[14]  Anna Alperovich,et al.  Reflection Separation in Light Fields based on Sparse Coding and Specular Flow , 2016, VMV.

[15]  Vladlen Koltun,et al.  A Simple Model for Intrinsic Image Decomposition with Depth Cues , 2013, 2013 IEEE International Conference on Computer Vision.

[16]  Jian Shi,et al.  Learning Non-Lambertian Object Intrinsics Across ShapeNet Categories , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Wei Yu,et al.  Neural EPI-Volume Networks for Shape from Light Field , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[18]  Ravi Ramamoorthi,et al.  Learning to Synthesize a 4D RGBD Light Field from a Single Image , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[19]  Marc Levoy,et al.  Light Fields and Computational Imaging , 2006, Computer.

[20]  Richard Szeliski,et al.  Extracting layers and analyzing their specular properties using epipolar-plane-image analysis , 2005, Comput. Vis. Image Underst..

[21]  Sven Wanner,et al.  Globally Consistent Multi-label Assignment on the Ray Space of 4D Light Fields , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  In-So Kweon,et al.  A Taxonomy and Evaluation of Dense Light Field Depth Estimation Algorithms , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[23]  Rob Fergus,et al.  Depth Map Prediction from a Single Image using a Multi-Scale Deep Network , 2014, NIPS.

[24]  H. Barrow,et al.  RECOVERING INTRINSIC SCENE CHARACTERISTICS FROM IMAGES , 1978 .

[25]  Marc Levoy,et al.  High performance imaging using large camera arrays , 2005, SIGGRAPH 2005.

[26]  Bastian Goldlücke,et al.  Accurate Depth and Normal Maps from Occlusion-Aware Focal Stack Symmetry , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Noah Snavely,et al.  Intrinsic images in the wild , 2014, ACM Trans. Graph..

[28]  Jiajun Wu,et al.  Self-Supervised Intrinsic Image Decomposition , 2017, NIPS.

[29]  Jitendra Malik,et al.  Depth Estimation and Specular Removal for Glossy Surfaces Using Point and Line Consistency with Light-Field Cameras , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Edward H. Adelson,et al.  Ground truth dataset and baseline evaluations for intrinsic image algorithms , 2009, 2009 IEEE 12th International Conference on Computer Vision.