NTIRE 2020 Challenge on Spectral Reconstruction from an RGB Image

This paper reviews the second challenge on spectral reconstruction from RGB images, i.e., the recovery of whole- scene hyperspectral (HS) information from a 3-channel RGB image. As in the previous challenge, two tracks were provided: (i) a "Clean" track where HS images are estimated from noise-free RGBs, the RGB images are themselves calculated numerically using the ground-truth HS images and supplied spectral sensitivity functions (ii) a "Real World" track, simulating capture by an uncalibrated and unknown camera, where the HS images are recovered from noisy JPEG-compressed RGB images. A new, larger-than-ever, natural hyperspectral image data set is presented, containing a total of 510 HS images. The Clean and Real World tracks had 103 and 78 registered participants respectively, with 14 teams competing in the final testing phase. A description of the proposed methods, alongside their challenge scores and an extensive evaluation of top performing methods is also provided. They gauge the state-of-the-art in spectral reconstruction from an RGB image.

[1]  Jongmin Park,et al.  NTIRE 2020 Challenge on NonHomogeneous Dehazing , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[2]  Diganta Misra,et al.  Mish: A Self Regularized Non-Monotonic Neural Activation Function , 2019, ArXiv.

[3]  Jie Zhao,et al.  Residual Pixel Attention Network for Spectral Reconstruction from RGB Images , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[4]  Andy Lambrechts,et al.  A compact snapshot multispectral imager with a monolithically integrated per-pixel filter mosaic , 2014, Photonics West - Micro and Nano Fabricated Electromechanical and Optical Components.

[5]  Vassilis Athitsos,et al.  lambda-Net: Reconstruct Hyperspectral Images From a Snapshot Measurement , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[6]  Lei Zhang,et al.  Pixel-aware Deep Function-mixture Network for Spectral Super-Resolution , 2019, AAAI.

[7]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[8]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Atmadeep Banerjee,et al.  MXR-U-Nets for Real Time Hyperspectral Reconstruction , 2020, ArXiv.

[10]  Frank Hutter,et al.  Decoupled Weight Decay Regularization , 2017, ICLR.

[11]  Radu Timofte,et al.  NTIRE 2020 Challenge on Image Demoireing: Methods and Results , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[12]  Michael S. Brown,et al.  Training-Based Spectral Reconstruction from a Single RGB Image , 2014, ECCV.

[13]  Yun Fu,et al.  Image Super-Resolution Using Very Deep Residual Channel Attention Networks , 2018, ECCV.

[14]  Hitoshi Kiya,et al.  Super-Resolution Using Convolutional Neural Networks Without Any Checkerboard Artifacts , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[15]  Mohamed Sedky,et al.  RGB to Spectral Reconstruction via Learned Basis Functions and Weights , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[16]  In-So Kweon,et al.  CBAM: Convolutional Block Attention Module , 2018, ECCV.

[17]  Radu Timofte,et al.  PIRM2018 Challenge on Spectral Image Super-Resolution: Methods and Results , 2018, ECCV Workshops.

[18]  Chein-I. Chang.  Hyperspectral Data Exploitation: Theory and Applications , 2007 .

[19]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Enhua Wu,et al.  Squeeze-and-Excitation Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Hanseok Ko,et al.  NTIRE 2020 Challenge on Real-World Image Super-Resolution: Methods and Results , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[22]  Dong Liu,et al.  HSCNN+: Advanced CNN-Based Hyperspectral Recovery from RGB Images , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[23]  Yandong Guo,et al.  NTIRE 2020 Challenge on Perceptual Extreme Super-Resolution: Methods and Results , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[24]  Xiaoling Zhang,et al.  NTIRE 2020 Challenge on Real Image Denoising: Dataset, Methods and Results , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[25]  Danna Zhou,et al.  d. , 1934, Microbial pathogenesis.

[26]  Radu Timofte,et al.  NTIRE 2020 Challenge on Image and Video Deblurring , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[27]  Sabine Süsstrunk,et al.  Multi-modal Spectral Image Super-Resolution , 2018, ECCV Workshops.

[28]  Fatih Tanriverdi,et al.  Dual Snapshot Hyperspectral Imaging System for 41-Band Spectral Analysis and Stereo Reconstruction , 2019, ISVC.

[29]  Radu Timofte,et al.  In Defense of Shallow Learned Spectral Reconstruction from RGB Images , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[30]  S. H. Amirshahi,et al.  Reconstruction of reflectance spectra using weighted principal component analysis , 2008 .

[31]  Radu Timofte,et al.  NTIRE 2020 Challenge on Video Quality Mapping: Methods and Results , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[32]  Jason Yosinski,et al.  An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution , 2018, NeurIPS.

[33]  Rui Song,et al.  Adaptive Weighted Attention Network with Camera Spectral Sensitivity Prior for Spectral Reconstruction from RGB Images , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[34]  Tao Zhang,et al.  Hyperspectral Image Reconstruction Using Deep External and Internal Learning , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[35]  Elena Beletkaia,et al.  More Than Meets the Eye , 2020 .

[36]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Leon A. Gatys,et al.  A Neural Algorithm of Artistic Style , 2015, ArXiv.

[38]  Han Zhang,et al.  Self-Attention Generative Adversarial Networks , 2018, ICML.

[39]  Yi-Tun Lin,et al.  Exposure Invariance in Spectral Reconstruction from RGB Images , 2019, Color and Imaging Conference.

[40]  Antonio Robles-Kelly,et al.  Single Image Spectral Reconstruction for Multimedia Applications , 2015, ACM Multimedia.

[41]  Luc Van Gool,et al.  Seven Ways to Improve Example-Based Single Image Super Resolution , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Andrew L. Maas.  Rectifier Nonlinearities Improve Neural Network Acoustic Models , 2013 .

[43]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[44]  Radu Timofte,et al.  An efficient CNN for spectral reconstruction from RGB images , 2018, ArXiv.

[45]  Boaz Arad,et al.  Sparse Recovery of Hyperspectral Signal from Natural RGB Images , 2016, ECCV.

[46]  Leslie N. Smith,et al.  A disciplined approach to neural network hyper-parameters: Part 1 - learning rate, batch size, momentum, and weight decay , 2018, ArXiv.

[47]  Wei Liu,et al.  Gated Fusion Network for Single Image Dehazing , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[48]  Yun Fu,et al.  Residual Dense Network for Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[49]  Radu Timofte,et al.  NTIRE 2018 Challenge on Spectral Reconstruction from RGB Images , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[50]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[51]  Radu Timofte,et al.  PIRM2018 Challenge on Spectral Image Super-Resolution: Dataset and Study , 2019, ECCV Workshops.

[52]  Brian A. Wandell,et al.  Spatio-spectral reconstruction of the multispectral datacube using sparse recovery , 2008, 2008 15th IEEE International Conference on Image Processing.

[53]  Hao Wu,et al.  Mixed Precision Training , 2017, ICLR.

[54]  Jun Zhang,et al.  Hyperspectral imaging from a raw mosaic image with end-to-end learning. , 2020, Optics express.

[55]  K. Uma,et al.  Light Weight Residual Dense Attention Net for Spectral Reconstruction from RGB Images , 2020, ArXiv.

[56]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[57]  Konrad Schindler,et al.  Learned Spectral Super-Resolution , 2017, ArXiv.

[58]  Christian Ledig,et al.  Checkerboard artifact free sub-pixel convolution: A note on sub-pixel convolution, resize convolution and convolution resize , 2017, ArXiv.

[59]  Sitao Xiang,et al.  On the Effects of Batch and Weight Normalization in Generative Adversarial Networks , 2017 .

[60]  Lai-Man Po,et al.  Hierarchical Regression Network for Spectral Reconstruction from RGB Images , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[61]  Zhi Zhang,et al.  Bag of Tricks for Image Classification with Convolutional Neural Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[62]  Luc Van Gool,et al.  A+: Adjusted Anchored Neighborhood Regression for Fast Super-Resolution , 2014, ACCV.

[63]  Andy Lambrechts,et al.  A novel CMOS-compatible, monolithically integrated line-scan hyperspectral imager covering the VIS-NIR range , 2016, SPIE Commercial + Scientific Sensing and Imaging.

[64]  Radu Timofte,et al.  Towards Spectral Estimation from a Single RGB Image in the Wild , 2018, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).