LTNet: Light Transfer Network for Depth Guided Image Relighting

Relighting is an interesting yet challenging low-level vision problem, which aims to re-render the scene with new light sources. In this paper, we introduce LTNet, a novel framework for image relighting. Unlike previous methods, we propose to solve this challenging problem by decoupling the enhancement process. Specifically, we propose to train a network that focuses on learning light variations. Our key insight is that light variations are the critical information to be learned because the scene stays unchanged during the light transfer process. To this end, we employ a global residual connection and corresponding residual loss for capturing light variations. Experimental results show that the proposed method achieves better visual quality on the VIDIT dataset in the NTIRE2021 relighting challenge.

[1]  Radu Timofte,et al.  NTIRE 2021 Depth Guided Image Relighting Challenge , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[2]  Lei Zhang,et al.  Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising , 2016, IEEE Transactions on Image Processing.

[3]  Rynson W. H. Lau,et al.  FormResNet: Formatted Residual Learning for Image Restoration , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[4]  Nisarg A. Shah,et al.  MSR-Net: Multi-Scale Relighting Network for One-to-One Relighting , 2021, ArXiv.

[5]  Narendra Ahuja,et al.  Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Xin Tong,et al.  Deferred Neural Lighting: Free-viewpoint Relighting from Unstructured Photographs , 2020 .

[7]  Stephen Gould,et al.  Single image depth estimation from predicted semantic labels , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[8]  Mohinder Malhotra.  Single Image Haze Removal Using Dark Channel Prior , 2016 .

[9]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[10]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Kalyan Sunkavalli,et al.  Deep image-based relighting from optimal sparse samples , 2018, ACM Trans. Graph..

[12]  Peter-Pike J. Sloan.  Normal mapping for precomputed radiance transfer , 2006, I3D '06.

[13]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.

[14]  Wan-Chi Siu,et al.  Deep Relighting Networks for Image Light Source Manipulation , 2020, ArXiv.

[15]  Yu-Bin Yang,et al.  Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections , 2016, NIPS.

[16]  Yun Fu,et al.  Residual Dense Network for Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[17]  S HrishikeshP.,et al.  WDRN : A Wavelet Decomposed RelightNet for Image Relighting , 2020, ECCV Workshops.

[18]  Sabine Süsstrunk,et al.  VIDIT: Virtual Image Dataset for Illumination Transfer , 2020, ArXiv.

[19]  Luc Van Gool,et al.  Replacing Mobile Camera ISP with a Single Deep Learning Model , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[20]  Qing Wang,et al.  AIM 2020: Scene Relighting and Illumination Estimation Challenge , 2020, ECCV Workshops.

[21]  Yun-Ta Tsai,et al.  Single image portrait relighting , 2019, ACM Trans. Graph..

[22]  Lei Zhang,et al.  FFDNet: Toward a Fast and Flexible Solution for CNN-Based Image Denoising , 2017, IEEE Transactions on Image Processing.

[23]  Kilian Q. Weinberger,et al.  Snapshot Ensembles: Train 1, get M for free , 2017, ICLR.