DeepBinaryMask: Learning a Binary Mask for Video Compressive Sensing

In this paper, we propose a novel encoder-decoder neural network model referred to as DeepBinaryMask for video compressive sensing. In video compressive sensing one frame is acquired using a set of coded masks (sensing matrix) from which a number of video frames is reconstructed, equal to the number of coded masks. The proposed framework is an end-to-end model where the sensing matrix is trained along with the video reconstruction. The encoder learns the binary elements of the sensing matrix and the decoder is trained to recover the unknown video sequence. The reconstruction performance is found to improve when using the trained sensing mask from the network as compared to other mask designs such as random, across a wide variety of compressive sensing reconstruction algorithms. Finally, our analysis and discussion offers insights into understanding the characteristics of the trained mask designs that lead to the improved reconstruction quality.

[1]  Ran El-Yaniv,et al.  Binarized Neural Networks , 2016, NIPS.

[2]  Yann LeCun,et al.  Effiicient BackProp , 1996, Neural Networks: Tricks of the Trade.

[3]  Jian Wang,et al.  LiSens- A Scalable Architecture for Video Compressive Sensing , 2015, 2015 IEEE International Conference on Computational Photography (ICCP).

[4]  Bernhard Schölkopf,et al.  A Machine Learning Approach for Non-blind Image Deconvolution , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  R. DeVore,et al.  A Simple Proof of the Restricted Isometry Property for Random Matrices , 2008 .

[6]  Ayan Chakrabarti,et al.  Learning Sensor Multiplexing Design through Back-propagation , 2016, NIPS.

[7]  Klaus-Robert Müller,et al.  Efficient BackProp , 2012, Neural Networks: Tricks of the Trade.

[8]  Jean Ponce,et al.  Learning a convolutional neural network for non-uniform motion blur removal , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Aggelos K. Katsaggelos,et al.  Video Super-Resolution With Convolutional Neural Networks , 2016, IEEE Transactions on Computational Imaging.

[10]  Ce Liu,et al.  Deep Convolutional Neural Network for Image Deconvolution , 2014, NIPS.

[11]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[12]  Aswin C. Sankaranarayanan,et al.  CS-MUVI: Video compressive sensing for spatial-multiplexing cameras , 2012, 2012 IEEE International Conference on Computational Photography (ICCP).

[13]  Ting Sun,et al.  Single-pixel imaging via compressive sampling , 2008, IEEE Signal Process. Mag..

[14]  Guillermo Sapiro,et al.  Coded aperture compressive temporal imaging , 2013, Optics express.

[15]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[16]  Enhong Chen,et al.  Image Denoising and Inpainting with Deep Neural Networks , 2012, NIPS.

[17]  Michael Elad,et al.  Optimized Projections for Compressed Sensing , 2007, IEEE Transactions on Signal Processing.

[18]  Xin Yuan,et al.  Snapshot Compressed Sensing: Performance Bounds and Algorithms , 2018, IEEE Transactions on Information Theory.

[19]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[20]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Guillermo Sapiro,et al.  Compressive Sensing by Learning a Gaussian Mixture Model From Measurements , 2015, IEEE Transactions on Image Processing.

[22]  Yiming Pi,et al.  Optimized Projection Matrix for Compressive Sensing , 2010, EURASIP J. Adv. Signal Process..

[23]  Pascal Vincent,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010, J. Mach. Learn. Res..

[24]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[25]  Pavan K. Turaga,et al.  ReconNet: Non-Iterative Reconstruction of Images from Compressively Sensed Random Measurements , 2016, ArXiv.

[26]  Aggelos K. Katsaggelos,et al.  Deep fully-connected networks for video compressive sensing , 2016, Digit. Signal Process..

[27]  Chun-Liang Li,et al.  One Network to Solve Them All — Solving Linear Inverse Problems Using Deep Projection Models , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[28]  Aggelos K. Katsaggelos,et al.  Preconditioning for Underdetermined Linear Systems with Sparse Solutions , 2015, IEEE Signal Processing Letters.

[29]  Chiye Li,et al.  Single-shot compressed ultrafast photography at one hundred billion frames per second , 2014, Nature.

[30]  José M. Bioucas-Dias,et al.  A New TwIST: Two-Step Iterative Shrinkage/Thresholding Algorithms for Image Restoration , 2007, IEEE Transactions on Image Processing.

[31]  Shree K. Nayar,et al.  Efficient Space-Time Sampling with Pixel-Wise Coded Exposure for High-Speed Imaging , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[33]  Xin Yuan,et al.  Compressive Imaging Via One-Shot Measurements , 2018, 2018 IEEE International Symposium on Information Theory (ISIT).

[34]  Aggelos K. Katsaggelos,et al.  Construction of Incoherent Unit Norm Tight Frames With Application to Compressed Sensing , 2014, IEEE Transactions on Information Theory.

[35]  Guillermo Sapiro,et al.  Video Compressive Sensing Using Gaussian Mixture Models , 2014, IEEE Transactions on Image Processing.

[36]  Li Xu,et al.  Shepard Convolutional Neural Networks , 2015, NIPS.

[37]  Igor Carron,et al.  XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks , 2016 .

[38]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[39]  Shiguang Shan,et al.  Deep Network Cascade for Image Super-resolution , 2014, ECCV.

[40]  Aswin C. Sankaranarayanan,et al.  FPA-CS: Focal plane array-based compressive imaging in short-wave infrared , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Guido Schuster,et al.  High spatio-temporal resolution video with compressed sensing. , 2015, Optics express.

[42]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  J.A. Anderson,et al.  Neurocomputing: Foundations of Research@@@Neurocomputing 2: Directions for Research , 1992 .

[44]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[45]  Yongdong Zhang,et al.  DR2-Net: Deep Residual Reconstruction Network for Image Compressive Sensing , 2017, Neurocomputing.

[46]  Yoshua Bengio,et al.  BinaryNet: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1 , 2016, ArXiv.

[47]  Liang Wang,et al.  Bidirectional Recurrent Convolutional Networks for Multi-Frame Super-Resolution , 2015, NIPS.

[48]  Stefan Harmeling,et al.  Image denoising: Can plain neural networks compete with BM3D? , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[49]  David L Donoho,et al.  Compressed sensing , 2006, IEEE Transactions on Information Theory.

[50]  Honglak Lee,et al.  Adaptive Multi-Column Deep Neural Networks with Application to Robust Image Denoising , 2013, NIPS.

[51]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[52]  Rama Chellappa,et al.  P2C2: Programmable pixel compressive camera for high speed imaging , 2011, CVPR 2011.

[53]  Razvan Pascanu,et al.  On the difficulty of training recurrent neural networks , 2012, ICML.

[54]  Yoshua Bengio,et al.  Neural Networks with Few Multiplications , 2015, ICLR.

[55]  Yoshua Bengio,et al.  BinaryConnect: Training Deep Neural Networks with binary weights during propagations , 2015, NIPS.

[56]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[57]  Clément Farabet,et al.  Torch7: A Matlab-like Environment for Machine Learning , 2011, NIPS 2011.

[58]  Qionghai Dai,et al.  Rank Minimization for Snapshot Compressive Imaging , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[59]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[60]  Richard G. Baraniuk,et al.  A deep learning approach to structured signal recovery , 2015, 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[61]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.