Supplementary File: LightPainter: Interactive Portrait Relighting with Freehand Scribble

Figure 5. Limitations of our method. While novice users can easily transfer the lighting patterns from a target image by simple scribbling, the scribble-based interface does not allow professional users to specify the exact shape/boundary of shadows. In LightPainter, these details are handled by the network to tolerate imperfect scribbles from novices. In addition, as shown in the relit image, our network also failed to produce non-Lambertian reflections in the subject’s eyes.

[1]  J. Kautz,et al.  Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation , 2022, ACM Trans. Graph..

[2]  Tao Yu,et al.  Geometry-aware Single-image Full-body Human Relighting , 2022, ECCV.

[3]  Michael J. Black,et al.  Towards Racially Unbiased Skin Tone Estimation via Scene Disambiguation , 2022, ECCV.

[4]  M. Sarkis,et al.  Face Relighting with Geometrically Consistent Shadows , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Taehyun Rhee,et al.  Deep Portrait Delighting , 2022, ECCV.

[6]  Vishal M. Patel,et al.  SketchEdit: Mask-Free Local Image Manipulation with Partial Sketches , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Ignas Budvytis,et al.  Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimation , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[8]  Xiaoguang Han,et al.  SketchHairSalon , 2021, ACM Trans. Graph..

[9]  Michel Sarkis,et al.  Towards High Fidelity Face Relighting with Realistic Shadows , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Jingyi Yu,et al.  Neural Video Portrait Relighting in Real-time via Consistency Modeling , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[11]  Ning Xu,et al.  Mask Guided Matting via Progressive Refinement Network , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Quan Wang,et al.  Single image portrait relighting via explicit multiple reflectance channel modeling , 2020, ACM Trans. Graph..

[13]  Yun-Ta Tsai,et al.  Neural Light Transport for Relighting and View Synthesis , 2020, ACM Transactions on Graphics.

[14]  Yun-Ta Tsai,et al.  Portrait shadow manipulation , 2020, ACM Trans. Graph..

[15]  Hao Li,et al.  Intuitive, Interactive Beard and Hair Synthesis With Generative Models , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Zhangyang Wang,et al.  Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches , 2020, ECCV.

[17]  David W. Jacobs,et al.  Deep Single-Image Portrait Relighting , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[18]  Alexei A. Efros,et al.  Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[19]  Andreas M. Lehrmann,et al.  Learning Physics-Guided Face Relighting Under Directional Light , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Yun-Ta Tsai,et al.  Single image portrait relighting , 2019, ACM Trans. Graph..

[21]  Mehran Ebrahimi,et al.  EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning , 2019, ArXiv.

[22]  Timo Aila,et al.  A Style-Based Generator Architecture for Generative Adversarial Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Thomas S. Huang,et al.  Free-Form Image Inpainting With Gated Convolution , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[24]  Matthias Zwicker,et al.  Faceshop , 2018, ACM Trans. Graph..

[25]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[26]  James Hays,et al.  SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[27]  Chuang Gan,et al.  Sparse, Smart Contours to Represent and Edit Images , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[28]  Carlos D. Castillo,et al.  SfSNet: Learning Shape, Reflectance and Illuminance of Faces 'in the Wild' , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[29]  Abhinav Gupta,et al.  Non-local Neural Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[30]  Sylvain Paris,et al.  Portrait lighting transfer using a mass transport approach , 2017, TOGS.

[31]  Ersin Yumer,et al.  Learning to predict indoor illumination from a single image , 2017, ACM Trans. Graph..

[32]  Eli Shechtman,et al.  StyLit , 2016, ACM Trans. Graph..

[33]  Vladlen Koltun,et al.  Multi-Scale Context Aggregation by Dilated Convolutions , 2015, ICLR.

[34]  Tieniu Tan,et al.  A Light CNN for Deep Face Representation With Noisy Labels , 2015, IEEE Transactions on Information Forensics and Security.

[35]  Jitendra Malik,et al.  Shape, Illumination, and Reflectance from Shading , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[37]  Volker Blanz,et al.  Realistic inverse lighting from a single 2D image of a face, taken under unknown and complex lighting , 2015, 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[38]  Lei Zhang,et al.  A Feature-Enriched Completely Blind Image Quality Evaluator , 2015, IEEE Transactions on Image Processing.

[39]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[40]  Kun Zhou,et al.  Intrinsic Face Image Decomposition with Human Face Priors , 2014, ECCV.

[41]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[42]  Frédo Durand,et al.  Style transfer for headshot portraits , 2014, ACM Trans. Graph..

[43]  Luc Van Gool,et al.  SEEDS: Superpixels Extracted Via Energy-Driven Sampling , 2012, International Journal of Computer Vision.

[44]  Shi-Min Hu,et al.  Sketch2Photo: internet image montage , 2009, ACM Trans. Graph..

[45]  Pieter Peers,et al.  Post-production facial performance relighting using reflectance transfer , 2007, ACM Trans. Graph..

[46]  Jun-Cheng Chen,et al.  An adaptive edge detection based colorization algorithm and its applications , 2005, ACM Multimedia.

[47]  Dani Lischinski,et al.  Colorization using optimization , 2004, ACM Trans. Graph..

[48]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[49]  Amnon Shashua,et al.  The Quotient Image: Class-Based Re-Rendering and Recognition with Varying Illuminations , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[50]  Paul E. Debevec,et al.  Acquiring the reflectance field of a human face , 2000, SIGGRAPH.

[51]  James H. Elder,et al.  Image editing in the contour domain , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[52]  Bui Tuong Phong.  Illumination for computer generated pictures , 1975, Commun. ACM.

[53]  P. Debevec,et al.  Total Relighting: Learning to Relight Portraits for Background Replacement , 2021 .

[54]  Ioannis A. Kakadiaris,et al.  Illumination-Invariant Face Recognition With Deep Relit Face Images , 2019, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[55]  Post-production Facial Performance Relighting Using Reflectance Transfer , 2022 .