Light Field Intrinsics with a Deep Encoder-Decoder Network

We present a fully convolutional autoencoder for light fields, which jointly encodes stacks of horizontal and vertical epipolar plane images through a deep network of residual layers. The complex structure of the light field is thus reduced to a comparatively low-dimensional representation, which can be decoded in a variety of ways. The different pathways of upconvolution we currently support are for disparity estimation and separation of the lightfield into diffuse and specular intrinsic components. The key idea is that we can jointly perform unsupervised training for the autoencoder path of the network, and supervised training for the other decoders. This way, we find features which are both tailored to the respective tasks and generalize well to datasets for which only example light fields are available. We provide an extensive evaluation on synthetic light field data, and show that the network yields good results on previously unseen real world data captured by a Lytro Illum camera and various gantries.

[1]  Anna Alperovich,et al.  Reflection Separation in Light Fields based on Sparse Coding and Specular Flow , 2016, VMV.

[2]  Katsushi Ikeuchi,et al.  Separating reflection components of textured surfaces using a single image , 2003, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Sven Wanner,et al.  Globally consistent depth labeling of 4D light fields , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Jian Shi,et al.  Learning Non-Lambertian Object Intrinsics Across ShapeNet Categories , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Reuben A. Farrugia,et al.  Light Field Compression With Homography-Based Low-Rank Approximation , 2017, IEEE Journal of Selected Topics in Signal Processing.

[7]  Marc Levoy,et al.  Light Fields and Computational Imaging , 2006, Computer.

[8]  Jitendra Malik,et al.  Depth Estimation and Specular Removal for Glossy Surfaces Using Point and Line Consistency with Light-Field Cameras , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Wei Yu,et al.  Neural EPI-Volume Networks for Shape from Light Field , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[10]  Taku Komura,et al.  A Deep Learning Framework for Character Motion Synthesis and Editing , 2016, ACM Trans. Graph..

[11]  Ravi Ramamoorthi,et al.  Learning to Synthesize a 4D RGBD Light Field from a Single Image , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[12]  Thomas Brox,et al.  Striving for Simplicity: The All Convolutional Net , 2014, ICLR.

[13]  Stefan B. Williams,et al.  Decoding, Calibration and Rectification for Lenselet-Based Plenoptic Cameras , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Takayuki Okatani,et al.  Separation of Reflection Components by Sparse Non-negative Matrix Factorization , 2014, ACCV.

[15]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[16]  Richard Szeliski,et al.  On the Motion and Appearance of Specularities in Image Sequences , 2002, ECCV.

[17]  Thomas Pock,et al.  Convolutional Networks for Shape from Light Field , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[19]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[20]  Hans-Peter Seidel,et al.  Gloss Editing in Light Fields , 2016, VMV.

[21]  Ashok Veeraraghavan,et al.  Light field denoising, light field superresolution and stereo camera based refocussing using a GMM light field patch prior , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[22]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[23]  Richard Szeliski,et al.  Extracting layers and analyzing their specular properties using epipolar-plane-image analysis , 2005, Comput. Vis. Image Underst..

[24]  Narendra Ahuja,et al.  Efficient and Robust Specular Highlight Removal , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Marcus A. Magnor,et al.  Data compression for light-field rendering , 2000, IEEE Trans. Circuits Syst. Video Technol..

[26]  Ting-Chun Wang,et al.  Learning-based view synthesis for light field cameras , 2016, ACM Trans. Graph..

[27]  Gordon Wetzstein,et al.  Compressive light field photography using overcomplete dictionaries and optimized projections , 2013, ACM Trans. Graph..

[28]  Sven Wanner,et al.  Reconstructing Reflective and Transparent Surfaces from Epipolar Plane Images , 2013, GCPR.

[29]  Joshua B. Tenenbaum,et al.  Deep Convolutional Inverse Graphics Network , 2015, NIPS.

[30]  Marc Levoy,et al.  High performance imaging using large camera arrays , 2005, ACM Trans. Graph..

[31]  Sebastian Nowozin,et al.  Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks , 2017, ICML.

[32]  Pavan K. Turaga,et al.  Compressive Light Field Reconstructions Using Deep Learning , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[33]  Sven Wanner,et al.  Datasets and Benchmarks for Densely Sampled 4D Light Fields , 2013, VMV.

[34]  In-So Kweon,et al.  Specular Reflection Separation Using Dark Channel Prior , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[35]  Edward H. Adelson,et al.  Ground truth dataset and baseline evaluations for intrinsic image algorithms , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[36]  In-So Kweon,et al.  Accurate depth map estimation from a lenslet light field camera , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Steven A. Shafer,et al.  Using color to separate reflection components , 1985 .

[38]  Jie Chen,et al.  Light Field Compression With Disparity-Guided Sparse Coding Based on Structural Key Views , 2016, IEEE Transactions on Image Processing.

[39]  Alexei A. Efros,et al.  A 4D Light-Field Dataset and CNN Architectures for Material Recognition , 2016, ECCV.

[40]  Katsushi Ikeuchi,et al.  Separating reflection components based on chromaticity and noise analysis , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  Alexei A. Efros,et al.  Occlusion-Aware Depth Estimation Using Light-Field Cameras , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[42]  Bastian Goldlücke,et al.  A Dataset and Evaluation Methodology for Depth Estimation on 4D Light Fields , 2016, ACCV.

[43]  Thomas Pock,et al.  Shape from Light Field Meets Robust PCA , 2014, ECCV.

[44]  Bastian Goldlücke,et al.  What Sparse Light Field Coding Reveals about Scene Structure , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  Bastian Goldlücke,et al.  Shadow and Specularity Priors for Intrinsic Light Field Decomposition , 2017, EMMCVPR.