Single-image Full-body Human Relighting

We present a single-image data-driven method to automatically relight images with full-body humans in them. Our framework is based on a realistic scene decomposition leveraging precomputed radiance transfer (PRT) and spherical harmonics (SH) lighting. In contrast to previous work, we lift the assumptions on Lambertian materials and explicitly model diffuse and specular reflectance in our data. Moreover, we introduce an additional light-dependent residual term that accounts for errors in the PRT-based image reconstruction. We propose a new deep learning architecture, tailored to the decomposition performed in PRT, that is trained using a combination of L1, logarithmic, and rendering losses. Our model outperforms the state of the art for full-body human relighting both with synthetic images and photographs.

[1]  Diego Gutierrez,et al.  The role of objective and subjective measures in material similarity learning , 2020, SIGGRAPH Posters.

[2]  Erik Reinhard,et al.  Light Source Detection in Photographs , 2009, CEIG.

[3]  Pratul P. Srinivasan,et al.  NeRF , 2020, ECCV.

[4]  Yun-Ta Tsai,et al.  Single image portrait relighting , 2019, ACM Trans. Graph..

[5]  Carlos D. Castillo,et al.  SfSNet: Learning Shape, Reflectance and Illuminance of Faces 'in the Wild' , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[6]  Ravi Ramamoorthi,et al.  What an image reveals about material reflectance , 2011, 2011 International Conference on Computer Vision.

[7]  Miguel A. Otaduy,et al.  An Appearance Model for Textile Fibers , 2017, Comput. Graph. Forum.

[8]  Kun Zhou,et al.  Intrinsic Light Field Images , 2016, Comput. Graph. Forum.

[9]  E. Land,et al.  Lightness and retinex theory. , 1971, Journal of the Optical Society of America.

[10]  V. S. Ramachandran,et al.  Perception of shape from shading , 1988, Nature.

[11]  Erik Reinhard,et al.  Multiple Light Source Estimation in a Single Image , 2013, Comput. Graph. Forum.

[12]  J. Koenderink,et al.  Light Source Dependence in Shape from Shading , 1997, Vision Research.

[13]  Qionghai Dai,et al.  Intrinsic video and applications , 2014, ACM Trans. Graph..

[14]  Kai Zhang,et al.  NeRF++: Analyzing and Improving Neural Radiance Fields , 2020, ArXiv.

[15]  Michael Goesele,et al.  Image-based rendering for scenes with reflections , 2012, ACM Trans. Graph..

[16]  Katsushi Ikeuchi,et al.  Numerical Shape from Shading and Occluding Boundaries , 1981, Artif. Intell..

[17]  Jan Kautz,et al.  Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments , 2002 .

[18]  Zhengqi Li,et al.  Learning Intrinsic Image Decomposition from Watching the World , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[19]  Jorge Jimenez.  Separable subsurface scattering , 2012, SIGGRAPH '12.

[20]  Ersin Yumer,et al.  Learning to predict indoor illumination from a single image , 2017, ACM Trans. Graph..

[21]  Diego Gutierrez,et al.  A similarity measure for material appearance , 2019, ACM Trans. Graph..

[22]  Adolfo Muñoz,et al.  Intrinsic Images by Clustering , 2012, Comput. Graph. Forum.

[23]  Alvaro Collet,et al.  High-quality streamable free-viewpoint video , 2015, ACM Trans. Graph..

[24]  Yun-Ta Tsai,et al.  Neural Light Transport for Relighting and View Synthesis , 2021, ACM Transactions on Graphics.

[25]  Mario Fritz,et al.  Reflectance and Natural Illumination from Single-Material Specular Objects Using Deep Learning , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Ko Nishino,et al.  Reflectance and Illumination Recovery in the Wild , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Bolei Zhou,et al.  Single Image Intrinsic Decomposition Without a Single Intrinsic Image , 2018, ECCV.

[28]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[29]  George Drettakis,et al.  Scalable inside-out image-based rendering , 2016, ACM Trans. Graph..

[30]  David W. Jacobs,et al.  Deep Single-Image Portrait Relighting , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[31]  Jonathan T. Barron,et al.  IBRNet: Learning Multi-View Image-Based Rendering , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Takayuki Okatani,et al.  Shape Reconstruction from an Endoscope Image by Shape from Shading Technique for a Point Light Source at the Projection Center , 1997, Comput. Vis. Image Underst..

[33]  Jean-François Lalonde,et al.  Learning Physics-Guided Face Relighting Under Directional Light , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Victor S. Lempitsky,et al.  Seamless Mosaicing of Image-Based Texture Maps , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[35]  Diego Gutierrez,et al.  A similarity measure for material appearance , 2019, ACM Trans. Graph..

[36]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Steve Marschner,et al.  Microfacet Models for Refraction through Rough Surfaces , 2007, Rendering Techniques.

[38]  Shree K. Nayar,et al.  Generalization of Lambert's reflectance model , 1994, SIGGRAPH.

[39]  Paul E. Debevec,et al.  Acquiring the reflectance field of a human face , 2000, SIGGRAPH.

[40]  Pat Hanrahan,et al.  An efficient representation for irradiance environment maps , 2001, SIGGRAPH.

[41]  Yaser Sheikh,et al.  3D object manipulation in a single photograph using stock 3D models , 2014, ACM Trans. Graph..

[42]  Jitendra Malik,et al.  Shape, Illumination, and Reflectance from Shading , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Jaakko Lehtinen,et al.  A framework for precomputed and captured light transport , 2007, TOGS.

[44]  Edward H. Adelson,et al.  Shape estimation in natural illumination , 2011, CVPR 2011.

[45]  Yair Weiss,et al.  Deriving intrinsic images from image sequences , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[46]  Jonathan T. Barron,et al.  NeRD: Neural Reflectance Decomposition from Image Collections , 2020, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[47]  Yannick Hold-Geoffroy,et al.  Deep Sky Modeling for Single Image Outdoor Lighting Estimation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Jitendra Malik,et al.  Color Constancy, Intrinsic Images, and Shape Estimation , 2012, ECCV.

[49]  Quan Wang,et al.  Single image portrait relighting via explicit multiple reflectance channel modeling , 2020, ACM Trans. Graph..

[50]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[51]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[52]  Harry Shum,et al.  Review of image-based rendering techniques , 2000, Visual Communications and Image Processing.

[53]  Yizhou Yu,et al.  Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping , 1998, Rendering Techniques.

[54]  Yannick Hold-Geoffroy,et al.  Deep Outdoor Illumination Estimation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[55]  Frank Hutter,et al.  Decoupled Weight Decay Regularization , 2017, ICLR.

[56]  Diego Gutierrez,et al.  The joint role of geometry and illumination on material recognition , 2021, Journal of vision.