Learnable Reconstruction Methods from RGB Images to Hyperspectral Imaging: A Survey

Hyperspectral imaging enables versatile applications due to its competence in capturing abundant spatial and spectral information, which are crucial for identifying substances. However, the devices for acquiring hyperspectral images are expensive and complicated. Therefore, many alternative spectral imaging methods have been proposed by directly reconstructing the hyperspectral information from lower-cost, more available RGB images. We present a thorough investigation of these stateof-the-art spectral reconstruction methods from the widespread RGB images. A systematic study and comparison of more than 25 methods has revealed that most of the data-driven deep learning methods are superior to prior-based methods in terms of reconstruction accuracy and quality despite lower speeds. This comprehensive review can serve as a fruitful reference source for peer researchers, thus further inspiring future development directions in related domains.

[1]  Jun Li,et al.  Accurate Spectral Super-resolution from Single RGB Image Using Multi-scale CNN , 2018, PRCV.

[2]  Yi Wang,et al.  Hyperspectral urban remote sensing image smoothing and enhancement using forward-and-backward diffusion , 2009, 2009 Joint Urban Remote Sensing Event.

[3]  Ayan Chakrabarti,et al.  Statistics of real-world hyperspectral images , 2011, CVPR 2011.

[4]  Yun Fu,et al.  Image Super-Resolution Using Very Deep Residual Channel Attention Networks , 2018, ECCV.

[5]  S. Silvestri,et al.  Mapping salt-marsh vegetation by multispectral and hyperspectral remote sensing , 2006 .

[6]  Lorenzo Bruzzone,et al.  Classification of hyperspectral remote sensing images with support vector machines , 2004, IEEE Transactions on Geoscience and Remote Sensing.

[7]  Dong Liu,et al.  HSCNN: CNN-Based Hyperspectral Image Recovery from Spectrally Undersampled Projections , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[8]  Tomas Pfister,et al.  Learning from Simulated and Unsupervised Images through Adversarial Training , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Joshua B. Broadwater,et al.  Tracking and Identification via Object Reflectance Using a Hyperspectral Video Camera , 2011 .

[10]  Joel A. Tropp,et al.  Simultaneous sparse approximation via greedy pursuit , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[11]  Jie Zhao,et al.  Residual Pixel Attention Network for Spectral Reconstruction from RGB Images , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[12]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[13]  Bahram Parvin,et al.  Classification of Histology Sections via Multispectral Convolutional Sparse Coding , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  S. Ustin,et al.  Mapping nonnative plants using hyperspectral imagery , 2003 .

[15]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[16]  R. Timofte,et al.  NTIRE 2018 Challenge on Spectral Reconstruction from RGB Images , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[17]  et al.,et al.  NTIRE 2020 Challenge on Spectral Reconstruction from an RGB Image , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[18]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[19]  Zhiwei Xiong,et al.  UDNet: Up-Down Network for Compact and Efficient Feature Representation in Image Super-Resolution , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[20]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[21]  Shutao Li,et al.  Hyperspectral Image Super-Resolution via Non-local Sparse Tensor Factorization , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[23]  Jon Atli Benediktsson,et al.  Advances in Hyperspectral Image Classification: Earth Monitoring with Statistical Learning Methods , 2013, IEEE Signal Processing Magazine.

[24]  Arif Mahmood,et al.  Hyperspectral Face Recognition With Spatiospectral Information Fusion and PLS Regression , 2015, IEEE Transactions on Image Processing.

[25]  Radu Timofte,et al.  An efficient CNN for spectral reconstruction from RGB images , 2018, ArXiv.

[26]  Atmadeep Banerjee,et al.  MXR-U-Nets for Real Time Hyperspectral Reconstruction , 2020, ArXiv.

[27]  Stephen Westland,et al.  Recovering spectral information using digital camera systems , 2001 .

[28]  Christian Ledig,et al.  Checkerboard artifact free sub-pixel convolution: A note on sub-pixel convolution, resize convolution and convolution resize , 2017, ArXiv.

[29]  Liang Gao,et al.  Snapshot Image Mapping Spectrometer (IMS) with high sampling density for hyperspectral microscopy , 2010, Optics express.

[30]  Ajmal Mian,et al.  Nonparametric Coupled Bayesian Dictionary and Classifier Learning for Hyperspectral Classification , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[31]  Kangfu Mei,et al.  Multi-scale Residual Network for Image Super-Resolution , 2018, ECCV.

[32]  Naveed Akhtar,et al.  Hyperspectral Recovery from RGB Images using Gaussian Processes , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  M. Borengasser,et al.  Hyperspectral Remote Sensing: Principles and Applications , 2007 .

[34]  Lei Zhang,et al.  Pixel-aware Deep Function-mixture Network for Spectral Super-Resolution , 2019, AAAI.

[35]  Pengfei Xiong,et al.  Pyramid Attention Network for Semantic Segmentation , 2018, BMVC.

[36]  Himanshu Sharma,et al.  2D-3D CNN Based Architectures for Spectral Reconstruction from RGB Images , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[37]  Xiaobai Sun,et al.  Video rate spectral imaging using a coded aperture snapshot spectral imager. , 2009, Optics express.

[38]  Masanori Suganuma,et al.  Dual Residual Networks Leveraging the Potential of Paired Operations for Image Restoration , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Mansour Moniri,et al.  Spectral-360: A Physics-Based Technique for Change Detection , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[40]  Xian-Hua Han,et al.  Residual HSRCNN: Residual Hyper-Spectral Reconstruction CNN from an RGB Image , 2018, 2018 24th International Conference on Pattern Recognition (ICPR).

[41]  John R. Miller,et al.  Hyperspectral vegetation indices and novel algorithms for predicting green LAI of crop canopies: Modeling and validation in the context of precision agriculture , 2004 .

[42]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[44]  Radu Timofte,et al.  Towards Spectral Estimation from a Single RGB Image in the Wild , 2018, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[45]  A. I. Negueruela,et al.  Use of three tristimulus values from surface reflectance spectra to calculate the principal components for reconstructing these spectra by using only three eigenvectors. , 2006, Journal of the Optical Society of America. A, Optics, image science, and vision.

[46]  Eustace L. Dereniak,et al.  Hyperspectral imaging for astronomy and space surviellance , 2004, SPIE Optics + Photonics.

[47]  Chong Wang,et al.  Locally Linear Embedded Sparse Coding for Spectral Reconstruction From RGB Images , 2018, IEEE Signal Processing Letters.

[48]  Lai-Man Po,et al.  Hierarchical Regression Network for Spectral Reconstruction from RGB Images , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[49]  Sune Svanberg,et al.  Multispectral system for medical fluorescence imaging , 1987 .

[50]  Radu Timofte,et al.  In Defense of Shallow Learned Spectral Reconstruction from RGB Images , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[51]  Yoichi Sato,et al.  Hyperspectral Image Super-Resolution With a Mosaic RGB Image , 2018, IEEE Transactions on Image Processing.

[52]  S. J.P..  Characteristic spectra of Munsell colors , 2002 .

[53]  Qian Du,et al.  Spatial Constrained Hyperspectral Reconstruction from RGB Inputs Using Dictionary Representation , 2019, IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium.

[54]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[55]  Malik Magdon-Ismail,et al.  On selecting a maximum volume sub-matrix of a matrix and related problems , 2009, Theor. Comput. Sci..

[56]  Jae Kwon Eem,et al.  Reconstruction of Surface Spectral Reflectances Using Characteristic Vectors of Munsell Colors , 1994, CIC.

[57]  John K. Tsotsos,et al.  From [R, G, B] to Surface Reflectance: Computing Color Constant Descriptors in Images , 1987, IJCAI.

[58]  K. Uma,et al.  Light Weight Residual Dense Attention Net for Spectral Reconstruction from RGB Images , 2020, ArXiv.

[59]  Huaici Zhao,et al.  Adversarial Networks for Scale Feature-Attention Spectral Image Reconstruction from a Single RGB , 2020, Sensors.

[60]  E. M. Winter,et al.  Anomaly detection from hyperspectral imagery , 2002, IEEE Signal Process. Mag..

[61]  DAVID ZHANG,et al.  A Comparative Study of Palmprint Recognition Algorithms , 2012, CSUR.

[62]  Hitoshi Kiya,et al.  Super-Resolution Using Convolutional Neural Networks Without Any Checkerboard Artifacts , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[63]  Christian Poellabauer,et al.  Second-Order Non-Local Attention Networks for Person Re-Identification , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[64]  Boaz Arad,et al.  Sparse Recovery of Hyperspectral Signal from Natural RGB Images , 2016, ECCV.

[65]  Enhua Wu,et al.  Squeeze-and-Excitation Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[66]  Joost van de Weijer,et al.  Adversarial Networks for Spatial Context-Aware Spectral Image Reconstruction from RGB , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[67]  Yoichi Sato,et al.  From RGB to Spectrum for Natural Scenes via Manifold-Based Mapping , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[68]  Guolan Lu,et al.  Medical hyperspectral imaging: a review , 2014, Journal of biomedical optics.

[69]  G J Edelman,et al.  Hyperspectral imaging for non-contact analysis of forensic traces. , 2012, Forensic science international.

[70]  Chuan Li,et al.  Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks , 2016, ECCV.

[71]  Han Zhang,et al.  Self-Attention Generative Adversarial Networks , 2018, ICML.

[72]  B A Wandell,et al.  Linear models of surface and illuminant spectra. , 1992, Journal of the Optical Society of America. A, Optics and image science.

[73]  Jeremy Howard,et al.  fastai: A Layered API for Deep Learning , 2020, Inf..

[74]  E. Cloutis,et al.  Review Article Hyperspectral geological remote sensing: evaluation of analytical techniques , 1996 .

[75]  Dorit Merhof,et al.  Reconstructing Spectral Images from RGB-Images Using a Convolutional Neural Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[76]  J. Cohen.  Dependency of the spectral reflectance curves of the Munsell color chips , 1964 .

[77]  Luc Van Gool,et al.  A+: Adjusted Anchored Neighborhood Regression for Fast Super-Resolution , 2014, ACCV.

[78]  Stephen Lin,et al.  Computational Snapshot Multispectral Cameras: Toward dynamic capture of the spectral world , 2016, IEEE Signal Processing Magazine.

[79]  Guillermo Sapiro,et al.  Discriminative sparse representations in hyperspectral imagery , 2010, 2010 IEEE International Conference on Image Processing.

[80]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[81]  Ute Beyer,et al.  Remote Sensing And Image Interpretation , 2016 .

[82]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[83]  Alfred S. McEwen,et al.  Spectral evidence for hydrated salts in recurring slope lineae on Mars , 2015 .

[84]  Mohamed Sedky,et al.  RGB to Spectral Reconstruction via Learned Basis Functions and Weights , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[85]  M. Sunshine,et al.  Spectral Analysis for Earth Science : Investigations Using Remote Sensing Data , 2013 .

[86]  Yoshua Bengio,et al.  The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[87]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[88]  Diganta Misra,et al.  Mish: A Self Regularized Non-Monotonic Neural Activation Function , 2019, ArXiv.

[89]  Jussi Parkkinen,et al.  Vector-subspace model for color representation , 1990 .

[90]  Jessika Weiss,et al.  Vision Science Photons To Phenomenology , 2016 .

[91]  Jason Yosinski,et al.  An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution , 2018, NeurIPS.

[92]  Yun Fu,et al.  Residual Dense Network for Image Restoration , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[93]  Arif Mahmood,et al.  Hyperspectral Face Recognition using 3D-DCT and Partial Least Squares , 2013, BMVC.

[94]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[95]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[96]  Konrad Schindler,et al.  Learned Spectral Super-Resolution , 2017, ArXiv.

[97]  Shu Wang,et al.  Multispectral Deep Neural Networks for Pedestrian Detection , 2016, BMVC.

[98]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[99]  Brian E. Smits.  An RGB-to-Spectrum Conversion for Reflectances , 1999, J. Graphics, GPU, & Game Tools.

[100]  Zhuowen Tu,et al.  On the Connection of Deep Fusion to Ensembling , 2016, ArXiv.

[101]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[102]  L. Maloney.  Evaluation of linear models of surface spectral reflectance with small numbers of parameters. , 1986, Journal of the Optical Society of America. A, Optics and image science.

[103]  Hua Huang,et al.  Spectral Reflectance Recovery From a Single RGB Image , 2018, IEEE Transactions on Computational Imaging.

[104]  Shree K. Nayar,et al.  Generalized Assorted Pixel Camera: Postcapture Control of Resolution, Dynamic Range, and Spectrum , 2010, IEEE Transactions on Image Processing.

[105]  Zhi Zhang,et al.  Bag of Tricks for Image Classification with Convolutional Neural Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[106]  Namil Kim,et al.  Multispectral pedestrian detection: Benchmark dataset and baseline , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[107]  Yi Yang,et al.  Attention to Scale: Scale-Aware Semantic Image Segmentation , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[108]  Wesley E. Snyder,et al.  Eigenviews for object recognition in multispectral imaging systems , 2003, 32nd Applied Imagery Pattern Recognition Workshop, 2003. Proceedings..

[109]  M. Descour,et al.  Computed-tomography imaging spectrometer: experimental calibration and reconstruction results. , 1995, Applied optics.

[110]  Jon Atli Benediktsson,et al.  Segmentation and classification of hyperspectral images using watershed transformation , 2010, Pattern Recognit..

[111]  Dong Liu,et al.  HSCNN+: Advanced CNN-Based Hyperspectral Recovery from RGB Images , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[112]  Rui Song,et al.  Adaptive Weighted Attention Network with Camera Spectral Sensitivity Prior for Spectral Reconstruction from RGB Images , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[113]  Bruno A. Olshausen,et al.  Learning Sparse Codes for Hyperspectral Imagery , 2011, IEEE Journal of Selected Topics in Signal Processing.

[114]  J. Chanussot,et al.  Hyperspectral Remote Sensing Data Analysis and Future Challenges , 2013, IEEE Geoscience and Remote Sensing Magazine.