NTIRE 2021 Depth Guided Image Relighting Challenge

Image relighting is attracting increasing interest due to its various applications. From a research perspective, im-age relighting can be exploited to conduct both image normalization for domain adaptation, and also for data augmentation. It also has multiple direct uses for photo montage and aesthetic enhancement. In this paper, we review the NTIRE 2021 depth guided image relighting challenge.We rely on the VIDIT dataset for each of our two challenge tracks, including depth information. The first track is on one-to-one relighting where the goal is to transform the illumination setup of an input image (color temperature and light source position) to the target illumination setup. In the second track, the any-to-any relighting challenge, the objective is to transform the illumination settings of the in-put image to match those of another guide image, similar to style transfer. In both tracks, participants were given depth information about the captured scenes. We had nearly 250 registered participants, leading to 18 confirmed team sub-missions in the final competition stage. The competitions, methods, and final results are presented in this paper.

[1]  A. N. Rajagopalan,et al.  Degradation Aware Approach to Image Restoration Using Knowledge Distillation , 2021, IEEE Journal of Selected Topics in Signal Processing.

[2]  Wan-Chi Siu,et al.  Deep Relighting Networks for Image Light Source Manipulation , 2020, ArXiv.

[3]  Alexandre Pierre Dherse,et al.  Scene relighting with illumination estimation in the latent space on an encoder-decoder scheme , 2020, ArXiv.

[4]  Kuan-Chih Huang,et al.  LAFFNet: A Lightweight Adaptive Feature Fusion Network for Underwater Image Enhancement , 2021, 2021 IEEE International Conference on Robotics and Automation (ICRA).

[5]  Huchuan Lu,et al.  Hierarchical Dynamic Filtering Network for RGB-D Salient Object Detection , 2020, ECCV.

[6]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[7]  Kai Zhao,et al.  Res2Net: A New Multi-Scale Backbone Architecture , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Maitreya Suin,et al.  Depth-Guided Dense Dynamic Filtering Network for Bokeh Effect Rendering , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[9]  Tao Lu,et al.  Multi-scale Self-calibrated Network for Image Light Source Transfer , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[10]  Chao-Han Huck Yang,et al.  Y-Net: Multi-Scale Feature Aggregation Network With Wavelet Structure Similarity Loss Function For Single Image Dehazing , 2020, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[11]  Qing Wang,et al.  SA-AE for Any-to-Any Relighting , 2020, ECCV Workshops.

[12]  Lai-Man Po,et al.  Hierarchical Regression Network for Spectral Reconstruction from RGB Images , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[13]  Mads Nielsen,et al.  Medical Image Computing and Computer-Assisted Intervention - MICCAI 2006, 9th International Conference, Copenhagen, Denmark, October 1-6, 2006, Proceedings, Part II , 2006, MICCAI.

[14]  Cong Leng,et al.  An Ensemble Neural Network for Scene Relighting with Light Classification , 2020, ECCV Workshops.

[15]  Jian-Jiun Ding,et al.  JSTASR: Joint Size and Transparency-Aware Snow Removal Algorithm Based on Modified Partial Convolution and Veiling Effect Removal , 2020, ECCV.

[16]  Daniel Cremers,et al.  FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-Based CNN Architecture , 2016, ACCV.

[17]  Xiaochun Cao,et al.  Single Image Dehazing via Multi-scale Convolutional Neural Networks , 2016, ECCV.

[18]  Kun Gao,et al.  NTIRE 2021 Learning the Super-Resolution Space Challenge , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[19]  Jian-Jiun Ding,et al.  PMS-Net: Robust Haze Removal Based on Patch Map for Single Images , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Majed El Helou,et al.  Stochastic Frequency Masking to Improve Super-Resolution and Denoising Networks , 2020, ECCV.

[21]  Luc Van Gool,et al.  Replacing Mobile Camera ISP with a Single Deep Learning Model , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[22]  S HrishikeshP.,et al.  WDRN : A Wavelet Decomposed RelightNet for Image Relighting , 2020, ECCV Workshops.

[23]  Sabine Süsstrunk,et al.  VIDIT: Virtual Image Dataset for Illumination Transfer , 2020, ArXiv.

[24]  Chunjie Zhang,et al.  LTNet: Light Transfer Network for Depth Guided Image Relighting , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[25]  Radu Timofte,et al.  NTIRE 2021 Challenge on Quality Enhancement of Compressed Video: Methods and Results , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[26]  Xiangyu Zhang,et al.  ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design , 2018, ECCV.

[27]  Sabine Süsstrunk,et al.  Correlation-based deblurring leveraging multispectral chromatic aberration in color and near-infrared joint acquisition , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[28]  Radu Timofte,et al.  NTIRE 2021 Challenge on Video Super-Resolution , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[29]  Shuhan Chen,et al.  Progressively Guided Alternate Refinement Network for RGB-D Salient Object Detection , 2020, ECCV.

[30]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[31]  Fahad Shahbaz Khan,et al.  NTIRE 2021 Challenge for Defocus Deblurring Using Dual-pixel Images: Methods and Results , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[32]  Nisarg A. Shah,et al.  MSR-Net: Multi-Scale Relighting Network for One-to-One Relighting , 2021, ArXiv.

[33]  Hao-Hsiang Yang,et al.  S3Net: A Single Stream Structure for Depth Guided Image Relighting , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[34]  Qing Wang,et al.  AIM 2020: Scene Relighting and Illumination Estimation Challenge , 2020, ECCV Workshops.

[35]  Paul Gafton,et al.  2D Image Relighting with Image-to-Image Translation , 2020, ArXiv.

[36]  Jian-Jiun Ding,et al.  PMHLD: Patch Map-Based Hybrid Learning DehazeNet for Single Image Haze Removal , 2020, IEEE Transactions on Image Processing.

[37]  Zhanwei Liu,et al.  Modified color CCD moiré method and its application in optical distortion correction , 2020 .

[38]  A. N. Rajagopalan,et al.  Spatially-Attentive Patch-Hierarchical Network for Adaptive Motion Deblurring , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Sabine Süsstrunk,et al.  A comparative study on wavelets and residuals in deep super resolution , 2019, Computational Imaging.

[40]  David W. Jacobs,et al.  Deep Single-Image Portrait Relighting , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[41]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Himanshu Kumar,et al.  DSRN: an Efficient Deep Network for Image Relighting , 2021, ICIP 2021.

[43]  Jonathan T. Barron,et al.  A General and Adaptive Robust Loss Function , 2017, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Radu Timofte,et al.  NTIRE 2021 Challenge on Perceptual Image Quality Assessment , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[45]  Radu Timofte,et al.  NTIRE 2021 Challenge on Image Deblurring , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[46]  Frederike Dümbgen,et al.  AAM: AN Assessment Metric of Axial Chromatic Aberration , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[47]  Hao-Hsiang Yang,et al.  Multi-modal Bifurcated Network for Depth Guided Image Relighting , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[48]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[49]  Vishal Monga,et al.  Physically Inspired Dense Fusion Networks for Relighting , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[50]  Jenq-Neng Hwang,et al.  NTIRE 2021 Multi-modal Aerial View Object Classification Challenge , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[51]  Sabine Süsstrunk,et al.  Blind Universal Bayesian Image Denoising With Gaussian Noise Level Learning , 2019, IEEE Transactions on Image Processing.

[52]  Radu Timofte,et al.  NTIRE 2021 Challenge on High Dynamic Range Imaging: Dataset, Methods and Results , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[53]  Bernard Llanos,et al.  Simultaneous Demosaicing and Chromatic Aberration Correction through Spectral Reconstruction , 2020, 2020 17th Conference on Computer and Robot Vision (CRV).

[54]  Yanyun Qu,et al.  Enhanced Pix2pix Dehazing Network , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[55]  James T. Kajiya,et al.  The rendering equation , 1986, SIGGRAPH.

[56]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[57]  Radu Timofte,et al.  NTIRE 2021 Challenge on Burst Super-Resolution: Methods and Results , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[58]  Ling Shao,et al.  NTIRE 2021 NonHomogeneous Dehazing Challenge Report , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).