SVBRDF-Invariant Shape and Reflectance Estimation from Light-Field Cameras

Light-field cameras have recently emerged as a powerful tool for one-shot passive 3D shape capture. However, obtaining the shape of glossy objects like metals, plastics or ceramics remains challenging, since standard Lambertian cues like photo-consistency cannot be easily applied. In this paper, we derive a spatially-varying (SV)BRDF-invariant theory for recovering 3D shape and reflectance from light-field cameras. Our key theoretical insight is a novel analysis of diffuse plus single-lobe SVBRDFs under a light-field setup. We show that, although direct shape recovery is not possible, an equation relating depths and normals can still be derived. Using this equation, we then propose using a polynomial (quadratic) shape prior to resolve the shape ambiguity. Once shape is estimated, we also recover the reflectance. We present extensive synthetic data on the entire MERL BRDF dataset, as well as a number of real examples to validate the theory, where we simultaneously recover shape and BRDFs from a single image taken with a Lytro Illum camera.

[1]  Alexei A. Efros,et al.  Occlusion-Aware Depth Estimation Using Light-Field Cameras , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[2]  Ronen Basri,et al.  From Shading to Local Shape , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  In-So Kweon,et al.  Accurate depth map estimation from a lenslet light field camera , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[5]  Adrien Treuille,et al.  Example-Based Stereo with General BRDFs , 2004, ECCV.

[6]  Alexei A. Efros,et al.  Depth Estimation with Occlusion Modeling Using Light-Field Cameras , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Todd E. Zickler,et al.  Blind Reflectometry , 2010, ECCV.

[8]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[9]  Ruigang Yang,et al.  Dealing with textureless regions and specular highlights - a progressive space carving scheme using a novel photo-consistency measure , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[10]  David J. Kriegman,et al.  Photometric stereo with non-parametric and spatially-varying reflectance , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Ravi Ramamoorthi,et al.  What an image reveals about material reflectance , 2011, 2011 International Conference on Computer Vision.

[12]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[13]  Peter F. Sturm,et al.  Voxel carving for specular surfaces , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[14]  Jitendra Malik,et al.  Depth Estimation and Specular Removal for Glossy Surfaces Using Point and Line Consistency with Light-Field Cameras , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Jitendra Malik,et al.  Depth from shading, defocus, and correspondence using light-field angular coherence , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Todd E. Zickler,et al.  Passive Reflectometry , 2008, ECCV.

[17]  Richard Szeliski,et al.  Manhattan-world stereo , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Wojciech Matusik,et al.  A data-driven reflectance model , 2003, ACM Trans. Graph..

[19]  Jean-Denis Durou,et al.  Numerical methods for shape-from-shading: A new survey with benchmarks , 2008, Comput. Vis. Image Underst..

[20]  Robert J. Woodham,et al.  Photometric method for determining surface orientation from multiple images , 1980 .

[21]  Manmohan Krishna Chandraker,et al.  What Camera Motion Reveals about Shape with Unknown BRDF , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Frédo Durand,et al.  Experimental analysis of BRDF models , 2005, EGSR '05.

[23]  Manmohan Krishna Chandraker,et al.  On Shape and Material Recovery from Motion , 2014, ECCV.

[24]  Jitendra Malik,et al.  Shape, albedo, and illumination from a single image of an unknown object , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[26]  Ko Nishino,et al.  Shape and Reflectance from Natural Illumination , 2012, ECCV.

[27]  Zhan Yu,et al.  Light Field Stereo Matching Using Bilateral Statistics of Surface Cameras , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[28]  Edward H. Adelson,et al.  Shape estimation in natural illumination , 2011, CVPR 2011.

[29]  Stefano Soatto,et al.  Multi-View Stereo Reconstruction of Dense Shape and Complex Appearance , 2005, International Journal of Computer Vision.

[30]  Manmohan Krishna Chandraker,et al.  The Information Available to a Moving Observer on Shape with Unknown, Isotropic BRDFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Lennart Wietzke,et al.  Single lens 3D-camera with extended depth-of-field , 2012, Electronic Imaging.

[32]  Jan-Michael Frahm,et al.  Piecewise planar and non-planar stereo for urban scene reconstruction , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[33]  Wojciech Matusik,et al.  Inverse shade trees for non-parametric material representation and editing , 2006, ACM Trans. Graph..

[34]  Steven M. Seitz,et al.  Schematic surface reconstruction , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[35]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[36]  Allan D. Jepson,et al.  Polynomial shape from shading , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[37]  Jitendra Malik,et al.  Depth Estimation for Glossy Surfaces with Light-Field Cameras , 2014, ECCV Workshops.

[38]  Alexei A. Efros,et al.  SVBRDF-Invariant Shape and Reflectance Estimation from a Light-Field Camera , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[39]  Shoji Tominaga,et al.  Surface Identification Using the Dichromatic Reflection Model , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[40]  Ko Nishino,et al.  Directional statistics-based reflectance model for isotropic bidirectional reflectance distribution functions. , 2011, Journal of the Optical Society of America. A, Optics, image science, and vision.

[41]  Sven Wanner,et al.  Globally consistent depth labeling of 4D light fields , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[42]  Jitendra Malik,et al.  Color Constancy, Intrinsic Images, and Shape Estimation , 2012, ECCV.

[43]  Ping-Sing Tsai,et al.  Shape from Shading: A Survey , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[44]  Tianli Yu,et al.  SDG Cut: 3D Reconstruction of Non-lambertian Objects Using Graph Cuts on Surface Distance Grid , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[45]  Jean Ponce,et al.  Accurate, Dense, and Robust Multiview Stereopsis , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  Yael Pritch,et al.  Scene reconstruction from high spatio-angular resolution light fields , 2013, ACM Trans. Graph..