Flexible SVBRDF Capture with a Multi‐Image Deep Network

Empowered by deep learning, recent methods for material capture can estimate a spatially‐varying reflectance from a single photograph. Such lightweight capture is in stark contrast with the tens or hundreds of pictures required by traditional optimization‐based approaches. However, a single image is often simply not enough to observe the rich appearance of real‐world materials. We present a deep‐learning method capable of estimating material appearance from a variable number of uncalibrated and unordered pictures captured with a handheld camera and flash. Thanks to an order‐independent fusing layer, this architecture extracts the most useful information from each picture, while benefiting from strong priors learned from data. The method can handle both view and light direction variation without calibration. We show how our method improves its prediction with the number of input pictures, and reaches high quality reconstructions with as little as 1 to 10 images ‐ a sweet spot between existing single‐image and complex multi‐image approaches.

[1]  Kalyan Sunkavalli,et al.  Materials for Masses: SVBRDF Acquisition with a Single Mobile Phone Image , 2018, ECCV.

[2]  Kun Zhou,et al.  Efficient reflectance capture using an autoencoder , 2018, ACM Trans. Graph..

[3]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[4]  Giuseppe Claudio Guarnera,et al.  BRDF Representation and Acquisition , 2016, Comput. Graph. Forum.

[5]  Michael Weinmann,et al.  Material Classification Based on Training Data Synthesized Using a BTF Database , 2014, ECCV.

[6]  Adrien Bousseau,et al.  Single-image SVBRDF capture with a rendering-aware deep network , 2018, ACM Trans. Graph..

[7]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[8]  Andrew Zisserman,et al.  SilNet : Single- and Multi-View Reconstruction by Learning from Silhouettes , 2017, BMVC.

[9]  John M. Snyder,et al.  Manifold bootstrapping for SVBRDF capture , 2010, ACM Trans. Graph..

[10]  Jaakko Lehtinen,et al.  Two-shot SVBRDF capture for stationary materials , 2015, ACM Trans. Graph..

[11]  Ersin Yumer,et al.  Material Editing Using a Physically Based Rendering Network , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[12]  Ko Nishino,et al.  Reflectance and Illumination Recovery in the Wild , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Frédo Durand,et al.  Burst Image Deblurring Using Permutation Invariant Convolutional Neural Networks , 2018, ECCV.

[14]  Shree K. Nayar,et al.  Reflectance and texture of real-world surfaces , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[15]  Andrew Gardner,et al.  Linear light source reflectometry , 2003, ACM Trans. Graph..

[16]  Robert L. Cook,et al.  A Reflectance Model for Computer Graphics , 1987, TOGS.

[17]  Pieter Peers,et al.  Appearance-from-motion , 2014, ACM Trans. Graph..

[18]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[19]  Kai Han,et al.  PS-FCN: A Flexible Learning Framework for Photometric Stereo , 2018, ECCV.

[20]  Mario Fritz,et al.  Reflectance and Natural Illumination from Single-Material Specular Objects Using Deep Learning , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Jason Yosinski,et al.  An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution , 2018, NeurIPS.

[22]  Baining Guo,et al.  Pocket reflectometry , 2011, SIGGRAPH 2011.

[23]  Kalyan Sunkavalli,et al.  Learning to reconstruct shape and spatially-varying reflectance from a single image , 2018, ACM Trans. Graph..

[24]  Andrew W. Fitzgibbon,et al.  BRDF and geometry capture from extended inhomogeneous samples using flash photography , 2005, Comput. Graph. Forum.

[25]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Pieter Peers,et al.  Mobile Surface Reflectometry , 2016, Comput. Graph. Forum.

[27]  Xiao Li,et al.  Single Image Surface Appearance Modeling with Self‐augmented CNNs and Inexact Supervision , 2018, Comput. Graph. Forum.

[28]  Abhijeet Ghosh,et al.  Polarization imaging reflectometry in the wild , 2017, ACM Trans. Graph..

[29]  Xiao Li,et al.  Modeling surface appearance from a single photograph using self-augmented convolutional neural networks , 2017, ACM Trans. Graph..

[30]  Alexander J. Smola,et al.  Deep Sets , 2017, 1703.06114.

[31]  Silvio Savarese,et al.  3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction , 2016, ECCV.

[32]  Steve Marschner,et al.  Estimating dual-scale properties of glossy surfaces from step-edge lighting , 2011, ACM Trans. Graph..

[33]  Anselmo Lastra,et al.  A generalized surface appearance representation for computer graphics , 2002 .

[34]  Jian Wang,et al.  Reflectance Capture Using Univariate Sampling of BRDFs , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[35]  Jaakko Lehtinen,et al.  Reflectance modeling by neural texture synthesis , 2016, ACM Trans. Graph..

[36]  Pieter Peers,et al.  Estimating Specular Roughness and Anisotropy from Second Order Spherical Gradient Illumination , 2009, Comput. Graph. Forum.