Tuning IR-cut Filter for Illumination-aware Spectral Reconstruction from RGB

To reconstruct spectral signals from multi-channel observations, in particular trichromatic RGBs, has recently emerged as a promising alternative to traditional scanning-based spectral imager. It has been proven that the reconstruction accuracy relies heavily on the spectral response of the RGB camera in use. To improve accuracy, data-driven algorithms have been proposed to retrieve the best response curves of existing RGB cameras, or even to design brand new three-channel response curves. Instead, this paper explores the filter-array based color imaging mechanism of existing RGB cameras, and proposes to design the IR-cut filter properly for improved spectral recovery, which stands out as an in-between solution with better trade-off between reconstruction accuracy and implementation complexity. We further propose a deep learning based spectral reconstruction method, which allows to recover the illumination spectrum as well. Experiment results with both synthetic and real images under daylight illumination have shown the benefits of our IR-cut filter tuning method and our illumination-aware spectral reconstruction method.

[1]  Zhenrong Zheng,et al.  Hyperspectral image recovery based on fusion of coded aperture snapshot spectral imaging and RGB images by guided filtering , 2020 .

[2]  Enhua Wu,et al.  Squeeze-and-Excitation Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Tao Zhang,et al.  Hyperspectral Image Reconstruction Using Deep External and Internal Learning , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[4]  A. Furchner,et al.  Hyperspectral infrared laser polarimetry for single-shot phase-amplitude imaging of thin films. , 2019, Optics letters.

[5]  Yidan Bao,et al.  Hyperspectral imaging for seed quality and safety inspection: a review , 2019, Plant Methods.

[6]  Masatoshi Okutomi,et al.  Single-Sensor RGB-NIR Imaging: High-Quality System Design and Prototype Implementation , 2019, IEEE Sensors Journal.

[7]  Tao Zhang,et al.  Joint Camera Spectral Sensitivity Selection and Hyperspectral Image Recovery , 2018, ECCV.

[8]  Xian-Hua Han,et al.  Residual HSRCNN: Residual Hyper-Spectral Reconstruction CNN from an RGB Image , 2018, 2018 24th International Conference on Pattern Recognition (ICPR).

[9]  Imari Sato,et al.  Deeply Learned Filter Response Functions for Hyperspectral Reconstruction , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[10]  Dong Liu,et al.  HSCNN+: Advanced CNN-Based Hyperspectral Recovery from RGB Images , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[11]  Radu Timofte,et al.  An efficient CNN for spectral reconstruction from RGB images , 2018, ArXiv.

[12]  Baohua Zhang,et al.  Hyperspectral Imaging and Their Applications in the Nondestructive Quality Assessment of Fruits and Vegetables , 2017, Hyperspectral Imaging in Agriculture, Food and Environment.

[13]  Boaz Arad,et al.  Filter Selection for Hyperspectral Estimation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[14]  Radu Timofte,et al.  In Defense of Shallow Learned Spectral Reconstruction from RGB Images , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[15]  Joost van de Weijer,et al.  Adversarial Networks for Spatial Context-Aware Spectral Image Reconstruction from RGB , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[16]  Yunsong Li,et al.  Hyperspectral image reconstruction by deep convolutional neural network for classification , 2017, Pattern Recognit..

[17]  Ze-Nian Li,et al.  Illumination and Reflectance Spectra Separation of Hyperspectral Image Data under Multiple Illumination Conditions , 2017, Color Imaging: Displaying, Processing, Hardcopy, and Applications.

[18]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Boaz Arad,et al.  Sparse Recovery of Hyperspectral Signal from Natural RGB Images , 2016, ECCV.

[20]  Jian Sun,et al.  Identity Mappings in Deep Residual Networks , 2016, ECCV.

[21]  Karla Hiller,et al.  Tunable MEMS Fabry-Pérot filters for infrared microspectrometers: a review , 2016, SPIE OPTO.

[22]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Yoichi Sato,et al.  Illumination and reflectance spectra separation of a hyperspectral image meets low-rank matrix factorization , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Roxana Savastru,et al.  Hyperspectral imaging-based wound analysis using mixture-tuned matched filtering classification method , 2015, Journal of biomedical optics.

[25]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[26]  Michael S. Brown,et al.  Training-Based Spectral Reconstruction from a Single RGB Image , 2014, ECCV.

[27]  Stephen Lin,et al.  Acquisition of High Spatial and Spectral Resolution Video with a Hybrid Camera System , 2014, International Journal of Computer Vision.

[28]  Michael W. Kudenov,et al.  Review of snapshot spectral imaging technologies , 2013, Optics and Precision Engineering.

[29]  Sabine Süsstrunk,et al.  What is the space of spectral sensitivity functions for digital color cameras? , 2013, 2013 IEEE Workshop on Applications of Computer Vision (WACV).

[30]  G J Edelman,et al.  Hyperspectral imaging for non-contact analysis of forensic traces. , 2012, Forensic science international.

[31]  Djemel Ziou,et al.  Image Quality Metrics: PSNR vs. SSIM , 2010, 2010 20th International Conference on Pattern Recognition.

[32]  Jon Atli Benediktsson,et al.  Segmentation and classification of hyperspectral images using watershed transformation , 2010, Pattern Recognit..

[33]  Neelam Gupta,et al.  Hyperspectral imager development at Army Research Laboratory , 2008, SPIE Defense + Commercial Sensing.

[34]  Graham D Finlayson,et al.  Analytic solution for separating spectra into illumination and surface reflectance components. , 2007, Journal of the Optical Society of America. A, Optics, image science, and vision.

[35]  G. Finlayson,et al.  A Re-evaluation of Colour Constancy Algorithm Performance , 2006 .

[36]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[37]  E. Land,et al.  Lightness and retinex theory. , 1971, Journal of the Optical Society of America.