Hyperspectral Image Reconstruction Using Deep External and Internal Learning

To solve the low spatial and/or temporal resolution problem which the conventional hypelrspectral cameras often suffer from, coded snapshot hyperspectral imaging systems have attracted more attention recently. Recovering a hyperspectral image (HSI) from its corresponding coded image is an ill-posed inverse problem, and learning accurate prior of HSI is essential to solve this inverse problem. In this paper, we present an effective convolutional neural network (CNN) based method for coded HSI reconstruction, which learns the deep prior from the external dataset as well as the internal information of input coded image with spatial-spectral constraint. Our method can effectively exploit spatial-spectral correlation and sufficiently represent the variety nature of HSIs. Experimental results show our method outperforms the state-of-the-art methods under both comprehensive quantitative metrics and perceptive quality.

[1]  Jian Yang,et al.  MemNet: A Persistent Memory Network for Image Restoration , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[2]  Pavan K. Turaga,et al.  ReconNet: Non-Iterative Reconstruction of Images from Compressively Sensed Random Measurements , 2016, ArXiv.

[3]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[4]  Dong Liu,et al.  HSCNN: CNN-Based Hyperspectral Image Recovery from Spectrally Undersampled Projections , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[5]  Liang Gao,et al.  Snapshot Image Mapping Spectrometer (IMS) with high sampling density for hyperspectral microscopy , 2010, Optics express.

[6]  Asgeir Bjorgan,et al.  Towards real-time medical diagnostics using hyperspectral imaging technology , 2015, European Conference on Biomedical Optics.

[7]  Yoichi Sato,et al.  Exploiting Spectral-Spatial Correlation for Coded Hyperspectral Image Restoration , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Yuehao Wu,et al.  Development of a digital micromirror device (DMD)-based Snapshot Spectral Imaging (DMD-SSI) system , 2011, IEEE Photonic Society 24th Annual Meeting.

[9]  Nahum Gat,et al.  Development of four-dimensional imaging spectrometers (4D-IS) , 2006, SPIE Optics + Photonics.

[10]  Ayan Chakrabarti,et al.  Statistics of real-world hyperspectral images , 2011, CVPR 2011.

[11]  Deyu Meng,et al.  Hyperspectral Image Classification With Markov Random Fields and a Convolutional Neural Network , 2017, IEEE Transactions on Image Processing.

[12]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[13]  Hideaki Haneishi,et al.  High-fidelity video and still-image communication based on spectral information: natural vision system and its applications , 2006, Electronic Imaging.

[14]  Alfred S. McEwen,et al.  Spectral evidence for hydrated salts in recurring slope lineae on Mars , 2015 .

[15]  Lucien Wald,et al.  Quality of high resolution synthesised images: Is there a simple criterion ? , 2000 .

[16]  Fred A. Kruse,et al.  The Spectral Image Processing System (SIPS) - Interactive visualization and analysis of imaging spectrometer data , 1993 .

[17]  Guangming Shi,et al.  Dual-camera design for coded aperture snapshot spectral imaging. , 2015, Applied optics.

[18]  José M. Bioucas-Dias,et al.  A New TwIST: Two-Step Iterative Shrinkage/Thresholding Algorithms for Image Restoration , 2007, IEEE Transactions on Image Processing.

[19]  I. Yamaguchi,et al.  Simultaneous acquisition of spectral image information. , 1991, Optics letters.

[20]  Giljoo Nam,et al.  High-quality hyperspectral reconstruction using a spectral prior , 2017, ACM Trans. Graph..

[21]  Qionghai Dai,et al.  Dual-coded compressive hyperspectral imaging. , 2014, Optics letters.

[22]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Guolan Lu,et al.  Medical hyperspectral imaging: a review , 2014, Journal of biomedical optics.

[24]  Guangming Shi,et al.  Adaptive Nonlocal Sparse Representation for Dual-Camera Compressive Hyperspectral Imaging , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Shree K. Nayar,et al.  Generalized Mosaicing: Wide Field of View Multispectral Imaging , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[27]  Gonzalo R. Arce,et al.  Compressive Hyperspectral Imaging via Approximate Message Passing , 2015, IEEE Journal of Selected Topics in Signal Processing.

[28]  Gang Sun,et al.  Squeeze-and-Excitation Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[29]  Qionghai Dai,et al.  Supplementary Document : Spatial-spectral Encoded Compressive Hyperspectral Imaging , 2014 .

[30]  M. Borengasser,et al.  Hyperspectral Remote Sensing: Principles and Applications , 2007 .

[31]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[32]  Boaz Arad,et al.  Sparse Recovery of Hyperspectral Signal from Natural RGB Images , 2016, ECCV.

[33]  Robert W. Basedow,et al.  HYDICE system: implementation and performance , 1995, Defense, Security, and Sensing.

[34]  Ashwin A. Wagadarikar,et al.  Single disperser design for coded aperture snapshot spectral imaging. , 2008, Applied optics.

[35]  M E Gehm,et al.  Single-shot compressive spectral imaging with a dual-disperser architecture. , 2007, Optics express.

[36]  Tao Zhang,et al.  HyperReconNet: Joint Coded Aperture Optimization and Image Reconstruction for Compressive Hyperspectral Imaging , 2019, IEEE Transactions on Image Processing.

[37]  Pavan K. Turaga,et al.  ReconNet: Non-Iterative Reconstruction of Images from Compressively Sensed Measurements , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Shree K. Nayar,et al.  Generalized Assorted Pixel Camera: Postcapture Control of Resolution, Dynamic Range, and Spectrum , 2010, IEEE Transactions on Image Processing.

[39]  Wallace M. Porter,et al.  A System Overview Of The Airborne Visible/Infrared Imaging Spectrometer (Aviris) , 1987, Optics & Photonics.

[40]  David J. Brady,et al.  Multiframe image estimation for coded aperture snapshot spectral imagers. , 2010, Applied optics.

[41]  Tao Zhang,et al.  Joint Camera Spectral Sensitivity Selection and Hyperspectral Image Recovery , 2018, ECCV.

[42]  Yun Fu,et al.  Image Super-Resolution Using Very Deep Residual Channel Attention Networks , 2018, ECCV.

[43]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[44]  Guangming Shi,et al.  High-speed hyperspectral video acquisition with a dual-camera architecture , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  M. Descour,et al.  Computed-tomography imaging spectrometer: experimental calibration and reconstruction results. , 1995, Applied optics.

[46]  M. Descour,et al.  Large-image-format computed tomography imaging spectrometer for fluorescence microscopy. , 2001, Optics express.