Appearance-from-motion

We present "appearance-from-motion", a novel method for recovering the spatially varying isotropic surface reflectance from a video of a rotating subject, with known geometry, under unknown natural illumination. We formulate the appearance recovery as an iterative process that alternates between estimating surface reflectance and estimating incident lighting. We characterize the surface reflectance by a data-driven microfacet model, and recover the microfacet normal distribution for each surface point separately from temporal changes in the observed radiance. To regularize the recovery of the incident lighting, we rely on the observation that natural lighting is sparse in the gradient domain. Furthermore, we exploit the sparsity of strong edges in the incident lighting to improve the robustness of the surface reflectance estimation. We demonstrate robust recovery of spatially varying isotropic reflectance from captured video as well as an internet video sequence for a wide variety of materials and natural lighting conditions.

[1]  Ko Nishino,et al.  Reflectance and Natural Illumination from a Single Image , 2012, ECCV.

[2]  Julie Dorsey,et al.  Digital Modeling of Material Appearance , 2007 .

[3]  John M. Snyder,et al.  Manifold bootstrapping for SVBRDF capture , 2010, ACM Trans. Graph..

[4]  Andrew Gardner,et al.  Linear light source reflectometry , 2003, ACM Trans. Graph..

[5]  C. Birklbauer,et al.  Light‐Field Retargeting , 2012, Comput. Graph. Forum.

[6]  Peter Shirley,et al.  A microfacet-based BRDF generator , 2000, SIGGRAPH.

[7]  Sunghyun Cho,et al.  Fast motion deblurring , 2009, SIGGRAPH 2009.

[8]  Adrien Treuille,et al.  Example-Based Stereo with General BRDFs , 2004, ECCV.

[9]  S. Marschner,et al.  Inverse Rendering for Computer Graphics , 1998 .

[10]  Tianli Yu,et al.  Sparse lumigraph relighting by illumination and reflectance estimation from multi-view images , 2006, EGSR '06.

[11]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Katsushi Ikeuchi,et al.  Determining Reflectance Parameters and Illumination Distribution from a Sparse Set of Images for View-dependent Image Synthesis , 2001, ICCV.

[13]  Steven M. Seitz,et al.  The Visual Turing Test for Scene Reconstruction , 2013, 2013 International Conference on 3D Vision.

[14]  Tianli Yu,et al.  Sparse lumigraph relighting by illumination and reflectance estimation from multi-view images , 2006, SIGGRAPH '06.

[15]  Baining Guo,et al.  Pocket reflectometry , 2011, SIGGRAPH 2011.

[16]  Qionghai Dai,et al.  Capturing Relightable Human Performances under General Uncontrolled Illumination , 2013, Comput. Graph. Forum.

[17]  F. E. Nicodemus,et al.  Geometrical considerations and nomenclature for reflectance , 1977 .

[18]  Li Xu,et al.  Two-Phase Kernel Estimation for Robust Motion Deblurring , 2010, ECCV.

[19]  Wolfgang Heidrich,et al.  Stochastic Deconvolution , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  Hans-Peter Seidel,et al.  Image-based reconstruction of spatial appearance and geometric detail , 2003, TOGS.

[21]  Todd E. Zickler,et al.  Blind Reflectometry , 2010, ECCV.

[22]  Sato Imari,et al.  Inverse Rendering for Computer Graphics , 2010 .

[23]  Hans-Peter Seidel,et al.  Relighting objects from image collections , 2009, CVPR 2009.

[24]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  Todd E. Zickler,et al.  A coaxial optical scanner for synchronous acquisition of 3D geometry and surface reflectance , 2010, SIGGRAPH 2010.

[26]  Steve Marschner,et al.  Estimating dual-scale properties of glossy surfaces from step-edge lighting , 2011, ACM Trans. Graph..

[27]  John Hart,et al.  ACM Transactions on Graphics , 2004, SIGGRAPH 2004.

[28]  Frédo Durand,et al.  Image and depth from a conventional camera with a coded aperture , 2007, ACM Trans. Graph..

[29]  Pat Hanrahan,et al.  A signal-processing framework for inverse rendering , 2001, SIGGRAPH.

[30]  Paul Graham,et al.  Acquiring reflectance and shape from continuous spherical harmonic illumination , 2013, ACM Trans. Graph..

[31]  Jaakko Lehtinen,et al.  Practical SVBRDF capture in the frequency domain , 2013, ACM Trans. Graph..

[32]  Baining Guo,et al.  Manifold bootstrapping for SVBRDF capture , 2010, SIGGRAPH 2010.

[33]  Jason Lawrence,et al.  A coaxial optical scanner for synchronous acquisition of 3D geometry and surface reflectance , 2010, ACM Transactions on Graphics.

[34]  Todd E. Zickler,et al.  Passive Reflectometry , 2008, ECCV.

[35]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[36]  Wojciech Matusik,et al.  A data-driven reflectance model , 2003, ACM Trans. Graph..

[37]  L. Rudin,et al.  Feature-oriented image enhancement using shock filters , 1990 .

[38]  Hans-Peter Seidel,et al.  Relighting objects from image collections , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[39]  Jitendra Malik,et al.  Shape, Illumination, and Reflectance from Shading , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Steven M. Seitz,et al.  Shape and materials by example: a photometric stereo approach , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..