Supplementary Material for “Self-supervised Neural Networks for Spectral Snapshot Compressive Imaging”

We consider using untrained neural networks to solve the reconstruction problem of snapshot compressive imaging (SCI), which uses a two-dimensional (2D) detector to capture a high-dimensional (usually 3D) data-cube in a compressed manner. Various SCI systems have been built in recent years to capture data such as high-speed videos, hyperspectral images, and the state-of-the-art reconstruction is obtained by the deep neural networks. However, most of these networks are trained in an end-to-end manner by a large amount of corpus with sometimes simulated ground truth, measurement pairs. In this paper, inspired by the untrained neural networks such as deep image priors (DIP) and deep decoders, we develop a framework by integrating DIP into the plug-and-play regime, leading to a self-supervised network for spectral SCI reconstruction. Extensive synthetic and real data results show that the proposed algorithm without training is capable of achieving competitive results to the training based networks. Furthermore, by integrating the proposed method with a pre-trained deep denoising prior, we have achieved state-of-the-art results. Our code is available at https://github.com/ mengziyi64/CASSI-Self-Supervised.

[1]  Xin Yuan,et al.  Snapshot Coherence Tomographic Imaging , 2021, IEEE Transactions on Computational Imaging.

[2]  Tao Zhang,et al.  Hyperspectral Image Reconstruction Using Deep External and Internal Learning , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[3]  Giljoo Nam,et al.  High-quality hyperspectral reconstruction using a spectral prior , 2017, ACM Trans. Graph..

[4]  Hua Huang,et al.  DNU: Deep Non-Local Unrolling for Computational Spectral Imaging , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Naoto Yokoya,et al.  Fast Hyperspectral Image Recovery via Non-iterative Fusion of Dual-Camera Compressive Hyperspectral Imaging , 2020, ArXiv.

[6]  Reinhard Heckel,et al.  Deep Decoder: Concise Image Representations from Untrained Non-convolutional Networks , 2018, ICLR.

[7]  Brendt Wohlberg,et al.  Plug-and-Play priors for model based reconstruction , 2013, 2013 IEEE Global Conference on Signal and Information Processing.

[8]  Qionghai Dai,et al.  Supplementary Document : Spatial-spectral Encoded Compressive Hyperspectral Imaging , 2014 .

[9]  Siming Zheng,et al.  Super-compression of large electron microscopy time series by deep compressive sensing learning , 2021, Patterns.

[10]  Shensheng Han,et al.  Deep plug-and-play priors for spectral snapshot compressive imaging , 2020, Photonics Research.

[11]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[12]  Xin Yuan,et al.  Perception Inspired Deep Neural Networks For Spectral Snapshot Compressive Imaging , 2021, 2021 IEEE International Conference on Image Processing (ICIP).

[13]  Mário A. T. Figueiredo,et al.  Gradient Projection for Sparse Reconstruction: Application to Compressed Sensing and Other Inverse Problems , 2007, IEEE Journal of Selected Topics in Signal Processing.

[14]  Qionghai Dai,et al.  Rank Minimization for Snapshot Compressive Imaging , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Ying Fu,et al.  Hyperspectral Image Reconstruction Using a Deep Spatial-Spectral Prior , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Xin Yuan,et al.  End-to-End Low Cost Compressive Spectral Imaging with Spatial-Spectral Self-Attention , 2020, ECCV.

[17]  José M. Bioucas-Dias,et al.  A New TwIST: Two-Step Iterative Shrinkage/Thresholding Algorithms for Image Restoration , 2007, IEEE Transactions on Image Processing.

[18]  Xin Yuan,et al.  Parallel lensless compressive imaging via deep convolutional neural networks. , 2018, Optics express.

[19]  Ulugbek S. Kamilov,et al.  RARE: Image Reconstruction Using Deep Priors Learned Without Groundtruth , 2019, IEEE Journal of Selected Topics in Signal Processing.

[20]  Guillermo Sapiro,et al.  Compressive Sensing by Learning a Gaussian Mixture Model From Measurements , 2015, IEEE Transactions on Image Processing.

[21]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[22]  G. Arce,et al.  LED-based compressive spectral-temporal imaging. , 2021, Optics express.

[23]  Shree K. Nayar,et al.  Multispectral Imaging Using Multiplexed Illumination , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[24]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[25]  William W. Hager,et al.  Updating the Inverse of a Matrix , 1989, SIAM Rev..

[26]  Charles A. Bouman,et al.  Plug-and-Play Priors for Bright Field Electron Tomography and Sparse Interpolation , 2015, IEEE Transactions on Computational Imaging.

[27]  Aggelos K. Katsaggelos,et al.  Snapshot Compressive Imaging: Theory, Algorithms, and Applications , 2021, IEEE Signal Processing Magazine.

[28]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[29]  Xin Yuan,et al.  BIRNAT: Bidirectional Recurrent Neural Networks with Adversarial Training for Video Snapshot Compressive Imaging , 2020, ECCV.

[30]  Bo Chen,et al.  Memory-Efficient Network for Large-scale Video Compressive Sensing , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[32]  Bo Chen,et al.  MetaSCI: Scalable and Adaptive Reconstruction for Video Compressive Sensing , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[34]  Guangming Shi,et al.  Adaptive Nonlocal Sparse Representation for Dual-Camera Compressive Hyperspectral Imaging , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Michael Elad,et al.  The Little Engine That Could: Regularization by Denoising (RED) , 2016, SIAM J. Imaging Sci..

[36]  Ashwin A. Wagadarikar,et al.  Single disperser design for coded aperture snapshot spectral imaging. , 2008, Applied optics.

[37]  Guillermo Sapiro,et al.  Coded aperture compressive temporal imaging , 2013, Optics express.

[38]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[39]  Guillermo Sapiro,et al.  Low-Cost Compressive Sensing for Color Video and Depth , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[40]  Jinli Suo,et al.  Plug-and-Play Algorithms for Video Snapshot Compressive Imaging , 2021, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  Xin Yuan,et al.  Generalized alternating projection based total variation minimization for compressive sensing , 2015, 2016 IEEE International Conference on Image Processing (ICIP).

[42]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  M E Gehm,et al.  Single-shot compressive spectral imaging with a dual-disperser architecture. , 2007, Optics express.

[44]  Boaz Arad,et al.  Sparse Recovery of Hyperspectral Signal from Natural RGB Images , 2016, ECCV.

[45]  Xin Yuan,et al.  Deep learning for video compressive sensing , 2020, APL Photonics.

[46]  Qionghai Dai,et al.  Plug-and-Play Algorithms for Large-Scale Snapshot Compressive Imaging , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[47]  Xin Yuan,et al.  Snapshot Compressed Sensing: Performance Bounds and Algorithms , 2018, IEEE Transactions on Information Theory.

[48]  David J. Brady,et al.  Multiframe image estimation for coded aperture snapshot spectral imagers. , 2010, Applied optics.

[49]  Zhenming Yu,et al.  Snapshot multispectral endomicroscopy. , 2020, Optics letters.

[50]  A. Ozcan,et al.  On the use of deep learning for computational imaging , 2019, Optica.

[51]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[52]  Dennis W Prather,et al.  Development of a digital-micromirror-device-based multishot snapshot spectral imaging system. , 2011, Optics letters.

[53]  Xin Yuan,et al.  Snapshot temporal compressive microscopy using an iterative algorithm with untrained neural networks. , 2021, Optics letters.

[54]  Vassilis Athitsos,et al.  lambda-Net: Reconstruct Hyperspectral Images From a Snapshot Measurement , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[55]  Michael Elad,et al.  DeepRED: Deep Image Prior Powered by RED , 2019, ICCV 2019.

[56]  Guangming Shi,et al.  Deep Gaussian Scale Mixture Prior for Spectral Compressive Imaging , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[57]  Xin Yuan,et al.  Compressive Hyperspectral Imaging With Side Information , 2015, IEEE Journal of Selected Topics in Signal Processing.

[58]  M. Elad,et al.  $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation , 2006, IEEE Transactions on Signal Processing.