Light stage super-resolution

The light stage has been widely used in computer graphics for the past two decades, primarily to enable the relighting of human faces. By capturing the appearance of the human subject under different light sources, one obtains the light transport matrix of that subject, which enables image-based relighting in novel environments. However, due to the finite number of lights in the stage, the light transport matrix only represents a sparse sampling on the entire sphere. As a consequence, relighting the subject with a point light or a directional source that does not coincide exactly with one of the lights in the stage requires interpolation and resampling the images corresponding to nearby lights, and this leads to ghosting shadows, aliased specularities, and other artifacts. To ameliorate these artifacts and produce better results under arbitrary high-frequency lighting, this paper proposes a learning-based solution for the "super-resolution" of scans of human faces taken from a light stage. Given an arbitrary "query" light direction, our method aggregates the captured images corresponding to neighboring lights in the stage, and uses a neural network to synthesize a rendering of the face that appears to be illuminated by a "virtual" light source at the query location. This neural network must circumvent the inherent aliasing and regularity of the light stage data that was used for training, which we accomplish through the use of regularized traditional interpolation methods within our network. Our learned model is able to produce renderings for arbitrary light directions that exhibit realistic shadows and specular highlights, and is able to generalize across a wide variety of subjects. Our super-resolution approach enables more accurate renderings of human subjects under detailed environment maps, or the construction of simpler light stages that contain fewer light sources while still yielding comparable quality renderings as light stages with more densely sampled lights.

[1]  Stephen Lin,et al.  Global illumination with radiance regression functions , 2013, ACM Trans. Graph..

[2]  Andrew Gardner,et al.  Performance relighting and reflectance transformation with time-multiplexed illumination , 2005, SIGGRAPH 2005.

[3]  Paul Graham,et al.  Acquiring reflectance and shape from continuous spherical harmonic illumination , 2013, ACM Trans. Graph..

[4]  Pieter Peers,et al.  Compressive light transport sensing , 2009, ACM Trans. Graph..

[5]  Ravi Ramamoorthi,et al.  Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines , 2019 .

[6]  Bruce H. Pillman Super-Resolution Imaging , 2013, J. Electronic Imaging.

[7]  Andrew Gardner,et al.  Performance relighting and reflectance transformation with time-multiplexed illumination , 2005, ACM Trans. Graph..

[8]  Pat Hanrahan,et al.  A signal-processing framework for inverse rendering , 2001, SIGGRAPH.

[9]  Paul E. Debevec,et al.  Acquiring the reflectance field of a human face , 2000, SIGGRAPH.

[10]  Carlos D. Castillo,et al.  SfSNet: Learning Shape, Reflectance and Illuminance of Faces 'in the Wild' , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[11]  Kalyan Sunkavalli,et al.  Deep image-based relighting from optimal sparse samples , 2018, ACM Trans. Graph..

[12]  Kun Zhou,et al.  Efficient reflectance capture using an autoencoder , 2018, ACM Trans. Graph..

[13]  Gordon Wetzstein,et al.  State of the Art on Neural Rendering , 2020, Comput. Graph. Forum.

[14]  Harry Shum,et al.  Plenoptic sampling , 2000, SIGGRAPH.

[15]  Soheil Darabi,et al.  Compressive Dual Photography , 2009, Comput. Graph. Forum.

[16]  Paul E. Debevec,et al.  The relightables , 2019, ACM Trans. Graph..

[17]  G. Rainer,et al.  Neural BTF Compression and Interpolation , 2019, Comput. Graph. Forum.

[18]  Matthew O'Toole,et al.  Optical computing for fast light transport analysis , 2010, ACM Trans. Graph..

[19]  Kaiming He,et al.  Group Normalization , 2018, ECCV.

[20]  Jaakko Lehtinen,et al.  E-LPIPS: Robust Perceptual Image Similarity via Random Transformation Ensembles , 2019, ArXiv.

[21]  Zhouchen Lin,et al.  Kernel Nyström method for light transport , 2009, ACM Trans. Graph..

[22]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[23]  Ting-Chun Wang,et al.  Learning-based view synthesis for light field cameras , 2016, ACM Trans. Graph..

[24]  Frédo Durand,et al.  A frequency analysis of light transport , 2005, SIGGRAPH '05.

[25]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[26]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[27]  Robert J. Woodham,et al.  Photometric method for determining surface orientation from multiple images , 1980 .

[28]  Katsushi Ikeuchi,et al.  Appearance sampling for obtaining a set of basis images for variable illumination , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[29]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[30]  Pat Hanrahan,et al.  All-frequency shadows using non-linear wavelet lighting approximation , 2003, ACM Trans. Graph..

[31]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[32]  Pieter Peers,et al.  Smooth Reconstruction and Compact Representation of Reflectance Functions for Image-based Relighting , 2004, Rendering Techniques.

[33]  Yaser Sheikh,et al.  Deep appearance models for face rendering , 2018, ACM Trans. Graph..

[34]  Paul E. Debevec,et al.  Practical Image-Based Relighting and Editing with Spherical-Harmonics and Local Lights , 2011, 2011 Conference for Visual Media Production.

[35]  Dong Liu,et al.  Light Field Super-Resolution: A Benchmark , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[36]  Kiriakos N. Kutulakos,et al.  Optical computing for fast light transport analysis , 2010, SIGGRAPH 2010.

[37]  Andrea Vedaldi,et al.  Instance Normalization: The Missing Ingredient for Fast Stylization , 2016, ArXiv.

[38]  Jaakko Lehtinen,et al.  Progressive Growing of GANs for Improved Quality, Stability, and Variation , 2017, ICLR.

[39]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[40]  Hans-Peter Seidel,et al.  Superresolution Reflectance Fields: Synthesizing images for intermediate light directions , 2007, Comput. Graph. Forum.

[41]  Paul Debevec,et al.  The Light Stages and Their Applications to Photoreal Digital Actors , 2012, SIGGRAPH 2012.

[42]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Peiran REN,et al.  Image based relighting using neural networks , 2015, ACM Trans. Graph..