Deeply Learned Filter Response Functions for Hyperspectral Reconstruction

Hyperspectral reconstruction from RGB imaging has recently achieved significant progress via sparse coding and deep learning. However, a largely ignored fact is that existing RGB cameras are tuned to mimic human trichromatic perception, thus their spectral responses are not necessarily optimal for hyperspectral reconstruction. In this paper, rather than use RGB spectral responses, we simultaneously learn optimized camera spectral response functions (to be implemented in hardware) and a mapping for spectral reconstruction by using an end-to-end network. Our core idea is that since camera spectral filters act in effect like the convolution layer, their response functions could be optimized by training standard neural networks. We propose two types of designed filters: a three-chip setup without spatial mosaicing and a single-chip setup with a Bayer-style 2x2 filter array. Numerical simulations verify the advantages of deeply learned spectral responses compared to existing RGB cameras. More interestingly, by considering physical restrictions in the design process, we are able to realize the deeply learned spectral response functions by using modern film filter production technologies, and thus construct data-inspired multispectral cameras for snapshot hyperspectral imaging.

[1]  Boaz Arad,et al.  Filter Selection for Hyperspectral Estimation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[2]  Dong Liu,et al.  HSCNN: CNN-Based Hyperspectral Image Recovery from Spectrally Undersampled Projections , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[3]  Liang Gao,et al.  Snapshot Image Mapping Spectrometer (IMS) with high sampling density for hyperspectral microscopy , 2010, Optics express.

[4]  Konrad Schindler,et al.  Hyperspectral Super-Resolution by Coupled Spectral Unmixing , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[5]  Stephen Lin,et al.  High resolution multispectral video capture with a hybrid camera system , 2011, CVPR 2011.

[6]  Ajmal S. Mian,et al.  Hierarchical Beta Process with Gaussian Process Prior for Hyperspectral Image Super Resolution , 2016, ECCV.

[7]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[8]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[9]  Guolan Lu,et al.  Medical hyperspectral imaging: a review , 2014, Journal of biomedical optics.

[10]  Yasuyuki Matsushita,et al.  High-resolution hyperspectral imaging via matrix factorization , 2011, CVPR 2011.

[11]  Yu-Wing Tai,et al.  RGB-Guided Hyperspectral Image Upsampling , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[12]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Shree K. Nayar,et al.  Generalized Assorted Pixel Camera: Postcapture Control of Resolution, Dynamic Range, and Spectrum , 2010, IEEE Transactions on Image Processing.

[14]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Ayan Chakrabarti,et al.  Statistics of real-world hyperspectral images , 2011, CVPR 2011.

[16]  Brian A. Wandell,et al.  Designing Illuminant Spectral Power Distributions for Surface Classification , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Ajmal S. Mian,et al.  Sparse Spatio-spectral Representation for Hyperspectral Image Super-resolution , 2014, ECCV.

[18]  Xiaobai Sun,et al.  Video rate spectral imaging using a coded aperture snapshot spectral imager. , 2009, Optics express.

[19]  Boaz Arad,et al.  Sparse Recovery of Hyperspectral Signal from Natural RGB Images , 2016, ECCV.

[20]  Joost van de Weijer,et al.  Adversarial Networks for Spatial Context-Aware Spectral Image Reconstruction from RGB , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[21]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  M. Eismann.  Hyperspectral Remote Sensing , 2012 .

[23]  Richard Bamler,et al.  A Sparse Image Fusion Algorithm With Application to Pan-Sharpening , 2013, IEEE Transactions on Geoscience and Remote Sensing.

[24]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Hideaki Matsuoka,et al.  Single-cell viability assessment with a novel spectro-imaging system. , 2002, Journal of biotechnology.

[26]  Ashok Veeraraghavan,et al.  ASP Vision: Optically Computing the First Layer of Convolutional Neural Networks Using Angle Sensitive Pixels , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[28]  Antonio Robles-Kelly,et al.  Single Image Spectral Reconstruction for Multimedia Applications , 2015, ACM Multimedia.

[29]  Yoichi Sato,et al.  From RGB to Spectrum for Natural Scenes via Manifold-Based Mapping , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[30]  Michael S. Brown,et al.  Training-Based Spectral Reconstruction from a Single RGB Image , 2014, ECCV.

[31]  Liangpei Zhang,et al.  A Practical Compressed Sensing-Based Pan-Sharpening Method , 2012, IEEE Geoscience and Remote Sensing Letters.

[32]  Konrad Schindler,et al.  Learned Spectral Super-Resolution , 2017, ArXiv.

[33]  Sabine Süsstrunk,et al.  What is the space of spectral sensitivity functions for digital color cameras? , 2013, 2013 IEEE Workshop on Applications of Computer Vision (WACV).