Discrete and Continuous-Time Soft-Thresholding for Dynamic Signal Recovery

There exist many well-established techniques to recover sparse signals from compressed measurements with known performance guarantees in the static case. However, only a few methods have been proposed to tackle the recovery of time-varying signals, and even fewer benefit from a theoretical analysis. In this paper, we study the capacity of the Iterative Soft-Thresholding Algorithm (ISTA) and its continuous-time analogue the Locally Competitive Algorithm (LCA) to perform this tracking in real time. ISTA is a well-known digital solver for static sparse recovery, whose iteration is a first-order discretization of the LCA differential equation. Our analysis shows that the outputs of both algorithms can track a time-varying signal while compressed measurements are streaming, even when no convergence criterion is imposed at each time step. The L2-distance between the target signal and the outputs of both discrete- and continuous-time solvers is shown to decay to a bound that is essentially optimal. Our analyses is supported by simulations on both synthetic and real data.

[1]  Christopher J. Rozell,et al.  Configurable hardware integrate and fire neurons for sparse approximation , 2013, Neural Networks.

[2]  Justin K. Romberg,et al.  Convergence of a neural network for sparse approximation using the nonsmooth Łojasiewicz inequality , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[3]  Yuriy V. Zakharov,et al.  DCD-RLS Adaptive Filters With Penalties for Sparse Identification , 2013, IEEE Transactions on Signal Processing.

[4]  Christopher J. Rozell,et al.  Dynamic filtering of sparse signals using reweighted ℓ1 , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[5]  Justin K. Romberg,et al.  Convergence Speed of a Dynamical System for Sparse Recovery , 2013, IEEE Transactions on Signal Processing.

[6]  Rebecca Willett,et al.  Dynamical Models and tracking regret in online convex programming , 2013, ICML.

[7]  Sergios Theodoridis,et al.  Generalized Thresholding and Online Sparsity-Aware Learning in a Union of Subspaces , 2011, IEEE Transactions on Signal Processing.

[8]  Christopher J. Rozell,et al.  A Common Network Architecture Efficiently Implements a Variety of Sparsity-Based Inference Problems , 2012, Neural Computation.

[9]  Christopher J. Rozell,et al.  Low Power Sparse Approximation on Reconfigurable Analog Hardware , 2012, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[10]  Daniel E. Quevedo,et al.  Compressive sampling for networked feedback control , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[11]  Justin K. Romberg,et al.  Convergence and Rate Analysis of Neural Networks for Sparse Approximation , 2011, IEEE Transactions on Neural Networks and Learning Systems.

[12]  Nazanin Rahnavard,et al.  TC-CSBP: Compressive sensing for time-correlated data based on belief propagation , 2011, 2011 45th Annual Conference on Information Sciences and Systems.

[13]  Philip Schniter,et al.  Tracking and smoothing of time-varying sparse signals via approximate belief propagation , 2010, 2010 Conference Record of the Forty Fourth Asilomar Conference on Signals, Systems and Computers.

[14]  Christophe Andrieu,et al.  Bayesian sequential compressed sensing in sparse dynamical systems , 2010, 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[15]  Vahid Tarokh,et al.  SPARLS: The Sparse RLS Algorithm , 2010, IEEE Transactions on Signal Processing.

[16]  Georgios B. Giannakis,et al.  Online Adaptive Estimation of Sparse Signals: Where RLS Meets the $\ell_1$ -Norm , 2010, IEEE Transactions on Signal Processing.

[17]  Sergios Theodoridis,et al.  Adaptive algorithm for sparse system identification using projections onto weighted ℓ1 balls , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[18]  Justin K. Romberg,et al.  Dynamic Updating for $\ell_{1}$ Minimization , 2009, IEEE Journal of Selected Topics in Signal Processing.

[19]  Tong Zhang.  Some sharp performance bounds for least squares regression with L1 regularization , 2009, 0908.2869.

[20]  G. Giannakis,et al.  Compressed sensing of time-varying signals , 2009, 2009 16th International Conference on Digital Signal Processing.

[21]  Marc Teboulle,et al.  A fast Iterative Shrinkage-Thresholding Algorithm with application to wavelet-based image deblurring , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[22]  Alfred O. Hero,et al.  Sparse LMS for system identification , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[23]  Yin Zhang,et al.  Fixed-Point Continuation for l1-Minimization: Methodology and Convergence , 2008, SIAM J. Optim..

[24]  Richard G. Baraniuk,et al.  Sparse Coding via Thresholding and Local Competition in Neural Circuits , 2008, Neural Computation.

[25]  Stephen J. Wright,et al.  Sparse Reconstruction by Separable Approximation , 2008, IEEE Transactions on Signal Processing.

[26]  Namrata Vaswani,et al.  Kalman filtered Compressed Sensing , 2008, 2008 15th IEEE International Conference on Image Processing.

[27]  J. Romberg,et al.  Imaging via Compressive Sampling , 2008, IEEE Signal Processing Magazine.

[28]  K. Bredies,et al.  Linear Convergence of Iterative Soft-Thresholding , 2007, 0709.1598.

[29]  José M. Bioucas-Dias,et al.  A New TwIST: Two-Step Iterative Shrinkage/Thresholding Algorithms for Image Restoration , 2007, IEEE Transactions on Image Processing.

[30]  Mário A. T. Figueiredo,et al.  Gradient Projection for Sparse Reconstruction: Application to Compressed Sensing and Other Inverse Problems , 2007, IEEE Journal of Selected Topics in Signal Processing.

[31]  Richard Baraniuk,et al.  The Dual-tree Complex Wavelet Transform , 2007 .

[32]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[33]  Andrzej Cichocki,et al.  Neural networks for optimization and signal processing , 1993 .