SVBRDF-Invariant Shape and Reflectance Estimation from a Light-Field Camera

Light-field cameras have recently emerged as a powerful tool for one-shot passive 3D shape capture. However, obtaining the shape of glossy objects like metals or plastics remains challenging, since standard Lambertian cues like photo-consistency cannot be easily applied. In this paper, we derive a spatially-varying (SV)BRDF-invariant theory for recovering 3D shape and reflectance from light-field cameras. Our key theoretical insight is a novel analysis of diffuse plus single-lobe SVBRDFs under a light-field setup. We show that, although direct shape recovery is not possible, an equation relating depths and normals can still be derived. Using this equation, we then propose using a polynomial (quadratic) shape prior to resolve the shape ambiguity. Once shape is estimated, we also recover the reflectance. We present extensive synthetic data on the entire MERL BRDF dataset, as well as a number of real examples to validate the theory, where we simultaneously recover shape and BRDFs from a single image taken with a Lytro Illum camera.

[1]  Alexei A. Efros,et al.  Occlusion-Aware Depth Estimation Using Light-Field Cameras , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[2]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[3]  Adrien Treuille,et al.  Example-Based Stereo with General BRDFs , 2004, ECCV.

[4]  Zhan Yu,et al.  Light Field Stereo Matching Using Bilateral Statistics of Surface Cameras , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Peter F. Sturm,et al.  Voxel carving for specular surfaces , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[6]  Jitendra Malik,et al.  Depth Estimation and Specular Removal for Glossy Surfaces Using Point and Line Consistency with Light-Field Cameras , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Jean-Denis Durou,et al.  Numerical methods for shape-from-shading: A new survey with benchmarks , 2008, Comput. Vis. Image Underst..

[8]  Alexei A. Efros,et al.  Depth Estimation with Occlusion Modeling Using Light-Field Cameras , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[10]  Ko Nishino,et al.  Directional statistics-based reflectance model for isotropic bidirectional reflectance distribution functions. , 2011, Journal of the Optical Society of America. A, Optics, image science, and vision.

[11]  Manmohan Krishna Chandraker,et al.  The Information Available to a Moving Observer on Shape with Unknown, Isotropic BRDFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Robert J. Woodham,et al.  Photometric method for determining surface orientation from multiple images , 1980 .

[13]  David J. Kriegman,et al.  Photometric stereo with non-parametric and spatially-varying reflectance , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Lennart Wietzke,et al.  Single lens 3D-camera with extended depth-of-field , 2012, Electronic Imaging.

[15]  Frédo Durand,et al.  Experimental analysis of BRDF models , 2005, EGSR '05.

[16]  Richard Szeliski,et al.  Manhattan-world stereo , 2009, CVPR.

[17]  Tianli Yu,et al.  SDG Cut: 3D Reconstruction of Non-lambertian Objects Using Graph Cuts on Surface Distance Grid , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[18]  Ko Nishino,et al.  Shape and Reflectance from Natural Illumination , 2012, ECCV.

[19]  Jitendra Malik,et al.  Depth Estimation for Glossy Surfaces with Light-Field Cameras , 2014, ECCV Workshops.

[20]  Stefano Soatto,et al.  Multi-View Stereo Reconstruction of Dense Shape and Complex Appearance , 2005, International Journal of Computer Vision.

[21]  Edward H. Adelson,et al.  Shape estimation in natural illumination , 2011, CVPR 2011.

[22]  Jan-Michael Frahm,et al.  Piecewise planar and non-planar stereo for urban scene reconstruction , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[23]  Shoji Tominaga,et al.  Surface Identification Using the Dichromatic Reflection Model , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[24]  Todd E. Zickler,et al.  Passive Reflectometry , 2008, ECCV.

[25]  Ronen Basri,et al.  From Shading to Local Shape , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  In-So Kweon,et al.  Accurate depth map estimation from a lenslet light field camera , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Ruigang Yang,et al.  Dealing with textureless regions and specular highlights - a progressive space carving scheme using a novel photo-consistency measure , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[28]  Wojciech Matusik,et al.  A data-driven reflectance model , 2003, ACM Trans. Graph..

[29]  Ravi Ramamoorthi,et al.  What an image reveals about material reflectance , 2011, 2011 International Conference on Computer Vision.

[30]  Allan D. Jepson,et al.  Polynomial shape from shading , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[31]  Jitendra Malik,et al.  Shape, albedo, and illumination from a single image of an unknown object , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[32]  Manmohan Krishna Chandraker,et al.  On Shape and Material Recovery from Motion , 2014, ECCV.

[33]  Jitendra Malik,et al.  Depth from shading, defocus, and correspondence using light-field angular coherence , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Alexei A. Efros,et al.  SVBRDF-Invariant Shape and Reflectance Estimation from Light-Field Cameras , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[36]  Sven Wanner,et al.  Globally consistent depth labeling of 4D light fields , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[37]  Jitendra Malik,et al.  Color Constancy, Intrinsic Images, and Shape Estimation , 2012, ECCV.

[38]  Ping-Sing Tsai,et al.  Shape from Shading: A Survey , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[39]  Manmohan Krishna Chandraker,et al.  What Camera Motion Reveals about Shape with Unknown BRDF , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[40]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[41]  Todd E. Zickler,et al.  Blind Reflectometry , 2010, ECCV.

[42]  Wojciech Matusik,et al.  Inverse shade trees for non-parametric material representation and editing , 2006, ACM Trans. Graph..

[43]  Steven M. Seitz,et al.  Schematic surface reconstruction , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[44]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[45]  Jean Ponce,et al.  Accurate, Dense, and Robust Multiview Stereopsis , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  Yael Pritch,et al.  Scene reconstruction from high spatio-angular resolution light fields , 2013, ACM Trans. Graph..