Discrete and Continuous-Time Soft-Thresholding for Dynamic Signal Recovery

There exist many well-established techniques to recover sparse signals from compressed measurements with known performance guarantees in the static case. More recently, new methods have been proposed to tackle the recovery of time-varying signals, but few benefit from a theoretical analysis. In this paper, we give theoretical guarantees for the Iterative Soft-Thresholding Algorithm (ISTA) and its continuous-time analogue the Locally Competitive Algorithm (LCA) to perform this tracking in real time. ISTA is a well-known digital solver for static sparse recovery, whose iteration is a first-order discretization of the LCA differential equation. Our analysis is based on the Restricted Isometry Property (RIP) and shows that the outputs of both algorithms can track a time-varying signal while compressed measurements are streaming, even when no convergence criterion is imposed at each time step. The l2-distance between the target signal and the outputs of both discrete- and continuous-time solvers is shown to decay to a bound that is essentially optimal. Our analysis is supported by simulations on both synthetic and real data.

[1]  Sergios Theodoridis,et al.  Generalized Thresholding and Online Sparsity-Aware Learning in a Union of Subspaces , 2011, IEEE Transactions on Signal Processing.

[2]  Christopher J. Rozell,et al.  Dynamic filtering of sparse signals using reweighted ℓ1 , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[3]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[4]  Justin K. Romberg,et al.  Convergence of a neural network for sparse approximation using the nonsmooth Łojasiewicz inequality , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[5]  Namrata Vaswani,et al.  Kalman filtered Compressed Sensing , 2008, 2008 15th IEEE International Conference on Image Processing.

[6]  Yin Zhang,et al.  Fixed-Point Continuation for l1-Minimization: Methodology and Convergence , 2008, SIAM J. Optim..

[7]  Rebecca Willett,et al.  Dynamical Models and tracking regret in online convex programming , 2013, ICML.

[8]  Andrzej Cichocki,et al.  Neural networks for optimization and signal processing , 1993 .

[9]  J. Romberg,et al.  Imaging via Compressive Sampling , 2008, IEEE Signal Processing Magazine.

[10]  Isao Yamada,et al.  A sparse adaptive filtering using time-varying soft-thresholding techniques , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[11]  Mário A. T. Figueiredo,et al.  Gradient Projection for Sparse Reconstruction: Application to Compressed Sensing and Other Inverse Problems , 2007, IEEE Journal of Selected Topics in Signal Processing.

[12]  José M. Bioucas-Dias,et al.  A New TwIST: Two-Step Iterative Shrinkage/Thresholding Algorithms for Image Restoration , 2007, IEEE Transactions on Image Processing.

[13]  K. Bredies,et al.  Linear Convergence of Iterative Soft-Thresholding , 2007, 0709.1598.

[14]  Yuriy V. Zakharov,et al.  DCD-RLS Adaptive Filters With Penalties for Sparse Identification , 2013, IEEE Transactions on Signal Processing.

[15]  Sergios Theodoridis,et al.  Adaptive algorithm for sparse system identification using projections onto weighted ℓ1 balls , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[16]  G. Giannakis,et al.  Compressed sensing of time-varying signals , 2009, 2009 16th International Conference on Digital Signal Processing.

[17]  Justin K. Romberg,et al.  Convergence Speed of a Dynamical System for Sparse Recovery , 2013, IEEE Transactions on Signal Processing.

[18]  Stephen J. Wright,et al.  Sparse Reconstruction by Separable Approximation , 2008, IEEE Transactions on Signal Processing.

[19]  Christopher J. Rozell,et al.  Configurable hardware integrate and fire neurons for sparse approximation , 2013, Neural Networks.

[20]  Richard Baraniuk,et al.  The Dual-tree Complex Wavelet Transform , 2007 .

[21]  Vahid Tarokh,et al.  SPARLS: The Sparse RLS Algorithm , 2010, IEEE Transactions on Signal Processing.

[22]  Laurent El Ghaoui,et al.  An Homotopy Algorithm for the Lasso with Online Observations , 2008, NIPS.

[23]  Christopher J. Rozell,et al.  A Common Network Architecture Efficiently Implements a Variety of Sparsity-Based Inference Problems , 2012, Neural Computation.

[24]  D. Donoho,et al.  Sparse MRI: The application of compressed sensing for rapid MR imaging , 2007, Magnetic resonance in medicine.

[25]  Christopher J. Rozell,et al.  Low Power Sparse Approximation on Reconfigurable Analog Hardware , 2012, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[26]  Daniel E. Quevedo,et al.  Compressive sampling for networked feedback control , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[27]  Christophe Andrieu,et al.  Bayesian sequential compressed sensing in sparse dynamical systems , 2010, 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[28]  Nazanin Rahnavard,et al.  TC-CSBP: Compressive sensing for time-correlated data based on belief propagation , 2011, 2011 45th Annual Conference on Information Sciences and Systems.

[29]  Philip Schniter,et al.  Tracking and smoothing of time-varying sparse signals via approximate belief propagation , 2010, 2010 Conference Record of the Forty Fourth Asilomar Conference on Signals, Systems and Computers.

[30]  Alfred O. Hero,et al.  Sparse LMS for system identification , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[31]  Richard G. Baraniuk,et al.  Sparse Coding via Thresholding and Local Competition in Neural Circuits , 2008, Neural Computation.

[32]  Georgios B. Giannakis,et al.  Online Adaptive Estimation of Sparse Signals: Where RLS Meets the $\ell_1$ -Norm , 2010, IEEE Transactions on Signal Processing.

[33]  Justin K. Romberg,et al.  Dynamic Updating for $\ell_{1}$ Minimization , 2009, IEEE Journal of Selected Topics in Signal Processing.

[34]  Justin K. Romberg,et al.  Convergence and Rate Analysis of Neural Networks for Sparse Approximation , 2011, IEEE Transactions on Neural Networks and Learning Systems.

[35]  Konstantinos Koutroumbas,et al.  A Variational Bayes Framework for Sparse Adaptive Estimation , 2014, IEEE Transactions on Signal Processing.

[36]  Marc Teboulle,et al.  A fast Iterative Shrinkage-Thresholding Algorithm with application to wavelet-based image deblurring , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[37]  Tong Zhang.  Some sharp performance bounds for least squares regression with L1 regularization , 2009, 0908.2869.