CS-Video : Algorithms , Architectures , and Applications for Compressive Video Sensing

The design of conventional sensors is based primarily on the Shannon-Nyquist sampling theorem, which states that a signal of bandwidth W Hz is fully determined by its discrete-time samples provided the sampling rate exceeds 2W samples per second. For discrete-time signals, the Shannon-Nyquist theorem has a very simple interpretation: the number of data samples must be at least as large as the dimensionality of the signal being sampled and recovered. This important result enables signal processing in the discrete-time domain without any loss of information. However, in an increasing number of applications, the Shannon-Nyquist sampling theorem dictates an unnecessary and often prohibitively high sampling rate. (See Box 1 for a derivation of the Nyquist rate of a time-varying scene.) As a motivating example, the high resolution of the image sensor hardware in modern cameras reflects the large amount of data sensed to capture an image. A 10-megapixel camera, in effect, takes 10 million measurements of the scene. Yet, almost immediately after acquisition, redundancies in the image are exploited to compress the acquired data significantly, often at compression ratios of 100:1 for visualization and even higher for detection and classification tasks. This example suggests immense wastage in the overall design of conventional cameras. Compressive sensing (CS) (see Box 2 and [6, 14, 16, 24]) is a powerful sensing paradigm that seeks to alleviate the daunting sampling rate requirements imposed by the Shannon-Nyquist principle. CS exploits the inherent structure (or redundancy) within the acquired signal to enable sampling and reconstruction at sub-Nyquist rates. The signal structure most commonly associated with CS is that of sparsity in a transform basis. This is the same structure exploited by image

[1]  R. Glowinski,et al.  Augmented Lagrangian and Operator-Splitting Methods in Nonlinear Mechanics , 1987 .

[2]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[3]  Richard G. Baraniuk,et al.  Optimal tree approximation with wavelets , 1999, Optics & Photonics.

[4]  Ronen Basri,et al.  Lambertian reflectance and linear subspaces , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[5]  David S. Taubman,et al.  Lifting-based invertible motion adaptive transform (LIMAT) framework for highly scalable video compression , 2003, IEEE Trans. Image Process..

[6]  Joel A. Tropp,et al.  Greed is good: algorithmic results for sparse approximation , 2004, IEEE Transactions on Information Theory.

[7]  E. Candès,et al.  New tight frames of curvelets and optimal representations of objects with piecewise C2 singularities , 2004 .

[8]  Lexing Ying,et al.  3D discrete curvelet transform , 2005, SPIE Optics + Photonics.

[9]  Steve Marschner,et al.  Dual photography , 2005, ACM Trans. Graph..

[10]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[11]  Amit K. Agrawal,et al.  Coded exposure photography: motion deblurring using fluttered shutter , 2006, ACM Trans. Graph..

[12]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[13]  Richard G. Baraniuk,et al.  Compressive imaging for video representation and coding , 2006 .

[14]  M. Elad,et al.  $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation , 2006, IEEE Transactions on Signal Processing.

[15]  Chinmay Hegde,et al.  Random Projections for Manifold Learning , 2007, NIPS.

[16]  Namrata Vaswani,et al.  Kalman filtered Compressed Sensing , 2008, 2008 15th IEEE International Conference on Image Processing.

[17]  Ashwin A. Wagadarikar,et al.  Single disperser design for coded aperture snapshot spectral imaging. , 2008, Applied optics.

[18]  M. Lustig,et al.  Compressed Sensing MRI , 2008, IEEE Signal Processing Magazine.

[19]  Mike E. Davies,et al.  Iterative Hard Thresholding for Compressed Sensing , 2008, ArXiv.

[20]  Stphane Mallat,et al.  A Wavelet Tour of Signal Processing, Third Edition: The Sparse Way , 2008 .

[21]  R. DeVore,et al.  A Simple Proof of the Restricted Isometry Property for Random Matrices , 2008 .

[22]  Volkan Cevher,et al.  Compressive Sensing for Background Subtraction , 2008, ECCV.

[23]  Rebecca Willett,et al.  Compressive coded aperture video reconstruction , 2008, 2008 16th European Signal Processing Conference.

[24]  Olgica Milenkovic,et al.  Subspace Pursuit for Compressive Sensing: Closing the Gap Between Performance and Complexity , 2008, ArXiv.

[25]  Ting Sun,et al.  Single-pixel imaging via compressive sampling , 2008, IEEE Signal Process. Mag..

[26]  E.J. Candes,et al.  An Introduction To Compressive Sampling , 2008, IEEE Signal Processing Magazine.

[27]  VershyninRoman,et al.  Uniform Uncertainty Principle and Signal Recovery via Regularized Orthogonal Matching Pursuit , 2009 .

[28]  Chun-Shien Lu,et al.  Distributed compressive video sensing , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[29]  Andrea Montanari,et al.  Message-passing algorithms for compressed sensing , 2009, Proceedings of the National Academy of Sciences.

[30]  Stephen J. Wright,et al.  Sparse Reconstruction by Separable Approximation , 2008, IEEE Transactions on Signal Processing.

[31]  Soheil Darabi,et al.  Compressive Dual Photography , 2009, Comput. Graph. Forum.

[32]  Marc Teboulle,et al.  Fast Gradient-Based Algorithms for Constrained Total Variation Image Denoising and Deblurring Problems , 2009, IEEE Transactions on Image Processing.

[33]  Deanna Needell,et al.  Uniform Uncertainty Principle and Signal Recovery via Regularized Orthogonal Matching Pursuit , 2007, Found. Comput. Math..

[34]  R. Raskar,et al.  Coded Strobing Photography : Compressive Sensing of High-speed Periodic Events , 2010 .

[35]  Tomoo Mitsunaga,et al.  Coded rolling shutter photography: Flexible space-time sampling , 2010, 2010 IEEE International Conference on Computational Photography (ICCP).

[36]  Wei Lu,et al.  Modified-CS: Modifying compressive sensing for problems with partially known support , 2009, 2009 IEEE International Symposium on Information Theory.

[37]  Tony F. Chan,et al.  A General Framework for a Class of First Order Primal-Dual Algorithms for Convex Optimization in Imaging Science , 2010, SIAM J. Imaging Sci..

[38]  Richard G. Baraniuk,et al.  Signal Processing With Compressive Measurements , 2010, IEEE Journal of Selected Topics in Signal Processing.

[39]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[40]  Deanna Needell,et al.  CoSaMP: Iterative signal recovery from incomplete and inaccurate samples , 2008, ArXiv.

[41]  Patrick Pérez,et al.  View-Independent Action Recognition from Temporal Self-Similarities , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Shree K. Nayar,et al.  Video from a single coded exposure photograph using a learned over-complete dictionary , 2011, 2011 International Conference on Computer Vision.

[43]  Aswin C. Sankaranarayanan,et al.  SpaRCS: Recovering low-rank and sparse matrices from compressive measurements , 2011, NIPS.

[44]  Rama Chellappa,et al.  P2C2: Programmable pixel compressive camera for high speed imaging , 2011, CVPR 2011.

[45]  Patrick L. Combettes,et al.  Proximal Splitting Methods in Signal Processing , 2009, Fixed-Point Algorithms for Inverse Problems in Science and Engineering.

[46]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[47]  Richard G. Baraniuk,et al.  VLSI Design of Approximate Message Passing for Signal Restoration and Compressive Sensing , 2012, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[48]  Aswin C. Sankaranarayanan,et al.  Flutter Shutter Video Camera for compressive sensing of videos , 2012, 2012 IEEE International Conference on Computational Photography (ICCP).

[49]  John Wright,et al.  Compressive principal component pursuit , 2012, 2012 IEEE International Symposium on Information Theory Proceedings.

[50]  Yin Zhang,et al.  A Compressive Sensing and Unmixing Scheme for Hyperspectral Data Processing , 2012, IEEE Transactions on Image Processing.

[51]  Elsa D. Angelini,et al.  Video reconstruction using compressed sensing measurements and 3d total variation regularization for bio-imaging applications , 2012, 2012 19th IEEE International Conference on Image Processing.

[52]  Richard G. Baraniuk,et al.  Regime Change: Bit-Depth Versus Measurement-Rate in Compressive Sensing , 2011, IEEE Transactions on Signal Processing.

[53]  Richard G. Baraniuk,et al.  The Pros and Cons of Compressive Sensing for Wideband Signal Acquisition: Noise Folding versus Dynamic Range , 2011, IEEE Transactions on Signal Processing.

[54]  Jean-Luc Starck,et al.  Sparse Solution of Underdetermined Systems of Linear Equations by Stagewise Orthogonal Matching Pursuit , 2012, IEEE Transactions on Information Theory.

[55]  James E. Fowler,et al.  Block-Based Compressed Sensing of Images and Video , 2012, Found. Trends Signal Process..

[56]  Michael B. Wakin,et al.  Multiscale algorithm for reconstructing videos from streaming compressive measurements , 2013, J. Electronic Imaging.

[57]  Gordon Wetzstein,et al.  Compressive light field photography using overcomplete dictionaries and optimized projections , 2013, ACM Trans. Graph..

[58]  Guillermo Sapiro,et al.  Coded aperture compressive temporal imaging , 2013, Optics express.

[59]  Richard G. Baraniuk,et al.  The STONE Transform: Multi-Resolution Image Enhancement and Real-Time Compressive Video , 2013, ArXiv.

[60]  M. Salman Asif,et al.  Motion‐adaptive spatio‐temporal regularization for accelerated dynamic MRI , 2013, Magnetic resonance in medicine.

[61]  Ashok Veeraraghavan,et al.  Towards Motion Aware Light Field Video for Dynamic Scenes , 2013, 2013 IEEE International Conference on Computer Vision.

[62]  Paul A. Wilford,et al.  Lensless imaging by compressive sensing , 2013, 2013 IEEE International Conference on Image Processing.

[63]  Thomas Burger,et al.  VLSI Design of a Monolithic Compressive-Sensing Wideband Analog-to-Information Converter , 2013, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[64]  Laurent Jacques,et al.  Robust 1-Bit Compressive Sensing via Binary Stable Embeddings of Sparse Vectors , 2011, IEEE Transactions on Information Theory.

[65]  Rama Chellappa,et al.  Compressive Acquisition of Linear Dynamical Systems , 2013, SIAM J. Imaging Sci..

[66]  Rebecca Willett,et al.  Compressive Coded Aperture Keyed Exposure Imaging with Optical Flow Reconstruction , 2013, ArXiv.

[67]  Richard G. Baraniuk,et al.  Fast Alternating Direction Optimization Methods , 2014, SIAM J. Imaging Sci..

[68]  Richard G. Baraniuk,et al.  A Field Guide to Forward-Backward Splitting with a FASTA Implementation , 2014, ArXiv.

[69]  A. Mahalanobis,et al.  Recent results of medium wave infrared compressive sensing. , 2014, Applied optics.

[70]  Richard G. Baraniuk,et al.  A deep learning approach to structured signal recovery , 2015, 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[71]  Min Li,et al.  Adaptive Primal-Dual Splitting Methods for Statistical Learning and Image Processing , 2015, NIPS.

[72]  Aswin C. Sankaranarayanan,et al.  FPA-CS: Focal plane array-based compressive imaging in short-wave infrared , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[73]  Massimo Fornasier,et al.  Compressive Sensing , 2015, Handbook of Mathematical Methods in Imaging.

[74]  Guido Schuster,et al.  High spatio-temporal resolution video with compressed sensing. , 2015, Optics express.

[75]  Jian Wang,et al.  LiSens- A Scalable Architecture for Video Compressive Sensing , 2015, 2015 IEEE International Conference on Computational Photography (ICCP).

[76]  Aswin C. Sankaranarayanan,et al.  Video Compressive Sensing for Spatial Multiplexing Cameras Using Motion-Flow Models , 2015, SIAM J. Imaging Sci..

[77]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[78]  Richard G. Baraniuk,et al.  Exponential Decay of Reconstruction Error From Binary Measurements of Sparse Signals , 2014, IEEE Transactions on Information Theory.

[79]  Aswin C. Sankaranarayanan,et al.  FlatCam: Thin, Lensless Cameras Using Coded Aperture and Computation , 2017, IEEE Transactions on Computational Imaging.

[80]  Rongrong Wang,et al.  From Compressed Sensing to Compressed Bit-Streams: Practical Encoders, Tractable Decoders , 2016, IEEE Transactions on Information Theory.