Multi-modal Bifurcated Network for Depth Guided Image Relighting

Image relighting aims to recalibrate the illumination setting in an image. In this paper, we propose a deep learning-based method called multi-modal bifurcated network (MB-Net) for depth guided image relighting. That is, given an image and the corresponding depth maps, a new image with the given illuminant angle and color temperature is generated by our network. This model extracts the image and the depth features by the bifurcated network in the encoder. To use the two features effectively, we adopt the dynamic dilated pyramid modules in the decoder. Moreover, to increase the variety of training data, we propose a novel data process pipeline to increase the number of the training data. Experiments conducted on the VIDIT dataset show that the proposed solution obtains the 1st place in terms of SSIM and PMS in the NTIRE 2021 Depth Guide One-to-one Relighting Challenge.

[1]  Qing Wang,et al.  AIM 2020: Scene Relighting and Illumination Estimation Challenge , 2020, ECCV Workshops.

[2]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[4]  Jonathan T. Barron,et al.  A General and Adaptive Robust Loss Function , 2017, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Jian-Jiun Ding,et al.  JSTASR: Joint Size and Transparency-Aware Snow Removal Algorithm Based on Modified Partial Convolution and Veiling Effect Removal , 2020, ECCV.

[6]  Sabine Süsstrunk,et al.  VIDIT: Virtual Image Dataset for Illumination Transfer , 2020, ArXiv.

[7]  Jie Liu,et al.  Asymmetric Two-Stream Architecture for Accurate RGB-D Saliency Detection , 2020, ECCV.

[8]  Qijun Zhao,et al.  RGB-D Salient Object Detection via 3D Convolutional Neural Networks , 2021, AAAI.

[9]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[10]  Xiaochun Cao,et al.  Single Image Dehazing via Multi-scale Convolutional Neural Networks , 2016, ECCV.

[11]  Dikpal Reddy,et al.  Frequency-Space Decomposition and Acquisition of Light Transport under Spatially Varying Illumination , 2012, ECCV.

[12]  Kalyan Sunkavalli,et al.  Deep image-based relighting from optimal sparse samples , 2018, ACM Trans. Graph..

[13]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[14]  Wojciech Matusik,et al.  Progressively-Refined Reflectance Functions from Natural Illumination , 2004 .

[15]  Jian-Jiun Ding,et al.  PMHLD: Patch Map-Based Hybrid Learning DehazeNet for Single Image Haze Removal , 2020, IEEE Transactions on Image Processing.

[16]  Lei Zhang,et al.  A Single Stream Network for Robust and Real-time RGB-D Salient Object Detection , 2020, ECCV.

[17]  Hao-Hsiang Yang,et al.  S3Net: A Single Stream Structure for Depth Guided Image Relighting , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[18]  S HrishikeshP.,et al.  WDRN : A Wavelet Decomposed RelightNet for Image Relighting , 2020, ECCV Workshops.

[19]  Hao-Hsiang Yang,et al.  Wavelet U-Net and the Chromatic Adaptation Transform for Single Image Dehazing , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[20]  Paul E. Debevec,et al.  Acquiring the reflectance field of a human face , 2000, SIGGRAPH.

[21]  Jian-Jiun Ding,et al.  PMS-Net: Robust Haze Removal Based on Patch Map for Single Images , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Radu Timofte,et al.  NTIRE 2021 Depth Guided Image Relighting Challenge , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[23]  Shuhan Chen,et al.  Progressively Guided Alternate Refinement Network for RGB-D Salient Object Detection , 2020, ECCV.

[24]  Sy-Yen Kuo,et al.  Color Channel-Based Smoke Removal Algorithm Using Machine Learning for Static Images , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[25]  Kuan-Chih Huang,et al.  LAFFNet: A Lightweight Adaptive Feature Fusion Network for Underwater Image Enhancement , 2021, 2021 IEEE International Conference on Robotics and Automation (ICRA).

[26]  Jan Kautz,et al.  Loss Functions for Image Restoration With Neural Networks , 2017, IEEE Transactions on Computational Imaging.

[27]  Huchuan Lu,et al.  Hierarchical Dynamic Filtering Network for RGB-D Salient Object Detection , 2020, ECCV.

[28]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[29]  Yu-Chiang Frank Wang,et al.  Wavelet Channel Attention Module With A Fusion Network For Single Image Deraining , 2020, 2020 IEEE International Conference on Image Processing (ICIP).

[30]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[31]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Sy-Yen Kuo,et al.  Efficient Reflection Removal Algorithm for Single Image by Pixel Compensation and Detail Reconstruction , 2018, 2018 IEEE 23rd International Conference on Digital Signal Processing (DSP).