Deep learning for video compressive sensing

We investigate deep learning for video compressive sensing within the scope of snapshot compressive imaging (SCI). In video SCI, multiple high-speed frames are modulated by different coding patterns and then a low-speed detector captures the integration of these modulated frames. In this manner, each captured measurement frame incorporates the information of all the coded frames, and reconstruction algorithms are then employed to recover the high-speed video. In this paper, we build a video SCI system using a digital micromirror device and develop both an end-to-end convolutional neural network (E2E-CNN) and a Plug-and-Play (PnP) framework with deep denoising priors to solve the inverse problem. We compare them with the iterative baseline algorithm GAP-TV and the state-of-the-art DeSCI on real data. Given a determined setup, a well-trained E2E-CNN can provide video-rate high-quality reconstruction. The PnP deep denoising method can generate decent results without task-specific pre-training and is faster than conventional iterative algorithms. Considering speed, accuracy, and flexibility, the PnP deep denoising method may serve as a baseline in video SCI reconstruction. To conduct quantitative analysis on these reconstruction algorithms, we further perform a simulation comparison on synthetic data. We hope that this study contributes to the applications of SCI cameras in our daily life.

[1]  Qionghai Dai,et al.  Rank Minimization for Snapshot Compressive Imaging , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Xin Yuan,et al.  High-speed compressive range imaging based on active illumination. , 2016, Optics express.

[3]  Guillermo Sapiro,et al.  Compressive Sensing by Learning a Gaussian Mixture Model From Measurements , 2015, IEEE Transactions on Image Processing.

[4]  Vinayak A. Rao,et al.  Hierarchical Infinite Divisibility for Multiscale Shrinkage , 2014, IEEE Transactions on Signal Processing.

[5]  Shree K. Nayar,et al.  Video from a single coded exposure photograph using a learned over-complete dictionary , 2011, 2011 International Conference on Computer Vision.

[6]  Michael Unser,et al.  Deep Convolutional Neural Network for Inverse Problems in Imaging , 2016, IEEE Transactions on Image Processing.

[7]  Xin Yuan,et al.  Compressive Hyperspectral Imaging With Side Information , 2015, IEEE Journal of Selected Topics in Signal Processing.

[8]  Guohai Situ,et al.  Deep-learning-based ghost imaging , 2017, Scientific Reports.

[9]  Ali Mousavi,et al.  Learning to invert: Signal recovery via Deep Convolutional Networks , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[10]  José M. Bioucas-Dias,et al.  A New TwIST: Two-Step Iterative Shrinkage/Thresholding Algorithms for Image Restoration , 2007, IEEE Transactions on Image Processing.

[11]  Xin Yuan,et al.  Parallel lensless compressive imaging via deep convolutional neural networks. , 2018, Optics express.

[12]  Shuai Li,et al.  Lensless computational imaging through deep learning , 2017, ArXiv.

[13]  Rama Chellappa,et al.  P2C2: Programmable pixel compressive camera for high speed imaging , 2011, CVPR 2011.

[14]  Vivek K Goyal,et al.  Quantum-inspired computational imaging , 2018, Science.

[15]  Lei Zhang,et al.  Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising , 2016, IEEE Transactions on Image Processing.

[16]  Xin Yuan,et al.  Compressive video sensing with side information. , 2017, Applied optics.

[17]  George Barbastathis,et al.  Low Photon Count Phase Retrieval Using Deep Learning. , 2018, Physical review letters.

[18]  A. Ozcan,et al.  On the use of deep learning for computational imaging , 2019, Optica.

[19]  Guillermo Sapiro,et al.  Adaptive temporal compressive sensing for video , 2013, 2013 IEEE International Conference on Image Processing.

[20]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Guillermo Sapiro,et al.  Video Compressive Sensing Using Gaussian Mixture Models , 2014, IEEE Transactions on Image Processing.

[22]  Xin Yuan,et al.  Compressive high-speed stereo imaging. , 2017, Optics express.

[23]  A. Maleki,et al.  From compression to compressed sensing , 2012, 2013 IEEE International Symposium on Information Theory.

[24]  Fengbo Ren,et al.  CSVideoNet: A Real-Time End-to-End Learning Framework for High-Frame-Rate Video Compressive Sensing , 2016, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[25]  Xin Yuan,et al.  Snapshot Compressed Sensing: Performance Bounds and Algorithms , 2018, IEEE Transactions on Information Theory.

[26]  Xin Yuan,et al.  Compressive video microscope via structured illumination , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[27]  Shiliang Sun,et al.  Multitask Twin Support Vector Machines , 2012, ICONIP.

[28]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[29]  Lawrence Carin,et al.  Spectral-temporal compressive imaging. , 2015, Optics letters.

[30]  Xin Yuan,et al.  Generalized alternating projection based total variation minimization for compressive sensing , 2015, 2016 IEEE International Conference on Image Processing (ICIP).

[31]  Lei Zhang,et al.  FFDNet: Toward a Fast and Flexible Solution for CNN-Based Image Denoising , 2017, IEEE Transactions on Image Processing.

[32]  Stephen Lin,et al.  Computational Snapshot Multispectral Cameras: Toward dynamic capture of the spectral world , 2016, IEEE Signal Processing Magazine.

[33]  Stanley H. Chan,et al.  Plug-and-Play ADMM for Image Restoration: Fixed-Point Convergence and Applications , 2016, IEEE Transactions on Computational Imaging.

[34]  Zhou Wang,et al.  Multiscale structural similarity for image quality assessment , 2003, The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003.

[35]  Aggelos K. Katsaggelos,et al.  Deep fully-connected networks for video compressive sensing , 2016, Digit. Signal Process..

[36]  Chun-Liang Li,et al.  One Network to Solve Them All — Solving Linear Inverse Problems Using Deep Projection Models , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[37]  Xiaobai Sun,et al.  Video rate spectral imaging using a coded aperture snapshot spectral imager. , 2009, Optics express.

[38]  Xin Yuan,et al.  Structured illumination temporal compressive microscopy. , 2016, Biomedical optics express.

[39]  Ashwin A. Wagadarikar,et al.  Single disperser design for coded aperture snapshot spectral imaging. , 2008, Applied optics.

[40]  Guillermo Sapiro,et al.  Coded aperture compressive temporal imaging , 2013, Optics express.

[41]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[42]  M E Gehm,et al.  Single-shot compressive spectral imaging with a dual-disperser architecture. , 2007, Optics express.