AIM 2020: Scene Relighting and Illumination Estimation Challenge

We review the AIM 2020 challenge on virtual image relighting and illumination estimation. This paper presents the novel VIDIT dataset used in the challenge and the different proposed solutions and final evaluation results over the 3 challenge tracks. The first track considered one-to-one relighting; the objective was to relight an input photo of a scene with a different color temperature and illuminant orientation (i.e., light source position). The goal of the second track was to estimate illumination settings, namely the color temperature and orientation, from a given image. Lastly, the third track dealt with any-to-any relighting, thus a generalization of the first track. The target color temperature and orientation, rather than being pre-determined, are instead given by a guide image. Participants were allowed to make use of their track 1 and 2 solutions for track 3. The tracks had 94, 52, and 56 registered participants, respectively, leading to 20 confirmed submissions in the final competition stage.

[1]  Xianming Liu,et al.  AIM 2019 Challenge on Video Extreme Super-Resolution: Methods and Results , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[2]  Chi-Wing Fu,et al.  Underexposed Photo Enhancement Using Deep Illumination Estimation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  S HrishikeshP.,et al.  WDRN : A Wavelet Decomposed RelightNet for Image Relighting , 2020, ECCV Workshops.

[4]  Radu Timofte,et al.  AIM 2020 Challenge on Image Extreme Inpainting , 2020, ECCV Workshops.

[5]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Noah Snavely,et al.  Intrinsic images in the wild , 2014, ACM Trans. Graph..

[7]  G. J. Burton,et al.  Color and spatial structure in natural scenes. , 1987, Applied optics.

[8]  Antonio Torralba,et al.  Statistics of natural image categories , 2003, Network.

[9]  David W. Jacobs,et al.  Deep Single-Image Portrait Relighting , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[10]  Radu Timofte,et al.  AIM 2020 Challenge on Rendering Realistic Bokeh , 2020, ECCV Workshops.

[11]  Jonathan T. Barron,et al.  Convolutional Color Constancy , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[12]  Mahmoud Afifi,et al.  What Else Can Fool Deep Learning? Addressing Color Constancy Errors on Deep Neural Network Performance , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[13]  Liang Lin,et al.  Multi-level Wavelet-CNN for Image Restoration , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[14]  Katsushi Ikeuchi,et al.  Illumination normalization with time-dependent intrinsic images for video surveillance , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Radu Timofte,et al.  AIM 2020 Challenge on Video Temporal Super-Resolution , 2020, ECCV Workshops.

[16]  Sylvain Paris,et al.  User-assisted intrinsic images , 2009, ACM Trans. Graph..

[17]  Radu Timofte,et al.  AIM 2020 Challenge on Learned Image Signal Processing Pipeline , 2020, ECCV Workshops.

[18]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Han Zhang,et al.  Self-Attention Generative Adversarial Networks , 2018, ICML.

[20]  Dan Xia,et al.  AIM 2020 Challenge on Video Extreme Super-Resolution: Methods and Results , 2020, ECCV Workshops.

[21]  Yair Weiss,et al.  Deriving intrinsic images from image sequences , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[22]  Balazs Kovacs,et al.  Shading Annotations in the Wild , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Wei Wei,et al.  AIM 2020 Challenge on Efficient Super-Resolution: Methods and Results , 2020, ECCV Workshops.

[24]  Jian Yang,et al.  Selective Kernel Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  D.Sabari Nathan,et al.  Moire Image Restoration using Multi Level Hyper Vision Net , 2020, ArXiv.

[26]  Cheng Lu,et al.  Intrinsic Images by Entropy Minimization , 2004, ECCV.

[27]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[28]  Naila Murray,et al.  Virtual KITTI 2 , 2020, ArXiv.

[29]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[30]  Jiawen Chen,et al.  Bilateral guided upsampling , 2016, ACM Trans. Graph..

[31]  In-So Kweon,et al.  CBAM: Convolutional Block Attention Module , 2018, ECCV.

[32]  Hao Li,et al.  Deep face normalization , 2019, ACM Trans. Graph..

[33]  Frederike Dümbgen,et al.  AAM: AN Assessment Metric of Axial Chromatic Aberration , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[34]  Zhanwei Liu,et al.  Modified color CCD moiré method and its application in optical distortion correction , 2020 .

[35]  Quoc V. Le,et al.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks , 2019, ICML.

[36]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[37]  Alexandre Pierre Dherse,et al.  Scene relighting with illumination estimation in the latent space on an encoder-decoder scheme , 2020, ArXiv.

[38]  Wan-Chi Siu,et al.  Deep Relighting Networks for Image Light Source Manipulation , 2020, ArXiv.

[39]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Miika Aittala,et al.  A Dataset of Multi-Illumination Images in the Wild , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[41]  Fahad Shahbaz Khan,et al.  AIM 2020 Challenge on Real Image Super-Resolution: Methods and Results , 2020, ECCV Workshops.

[42]  Sabine Süsstrunk,et al.  VIDIT: Virtual Image Dataset for Illumination Transfer , 2020, ArXiv.

[43]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[44]  Zhuowen Tu,et al.  Aggregated Residual Transformations for Deep Neural Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  Zhengqi Li,et al.  Learning Intrinsic Image Decomposition from Watching the World , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[46]  Andrea Vedaldi,et al.  Instance Normalization: The Missing Ingredient for Fast Stylization , 2016, ArXiv.

[47]  Michael S. Brown,et al.  When Color Constancy Goes Wrong: Correcting Improperly White-Balanced Images , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Michael S. Brown,et al.  Sensor-Independent Illumination Estimation for DNN Models , 2019, BMVC.

[49]  Kalyan Sunkavalli,et al.  Deep image-based relighting from optimal sparse samples , 2018, ACM Trans. Graph..

[50]  Yun-Ta Tsai,et al.  Single image portrait relighting , 2019, ACM Trans. Graph..

[51]  Edward H. Adelson,et al.  Recovering intrinsic images from a single image , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[52]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[53]  Jitendra Malik,et al.  Color Constancy, Intrinsic Images, and Shape Estimation , 2012, ECCV.

[54]  Bernard Llanos,et al.  Simultaneous Demosaicing and Chromatic Aberration Correction through Spectral Reconstruction , 2020, 2020 17th Conference on Computer and Robot Vision (CRV).

[55]  Xuelong Li,et al.  Intrinsic images using optimization , 2011, CVPR 2011.