Online Sparse System Identification and Signal Reconstruction Using Projections Onto Weighted $\ell_{1}$ Balls

This paper presents a novel projection-based adaptive algorithm for sparse signal and system identification. The sequentially observed data are used to generate an equivalent sequence of closed convex sets, namely hyperslabs. Each hyperslab is the geometric equivalent of a cost criterion, that quantifies “data mismatch.” Sparsity is imposed by the introduction of appropriately designed weighted ℓ1 balls and the related projection operator is also derived. The algorithm develops around projections onto the sequence of the generated hyperslabs as well as the weighted ℓ1 balls. The resulting scheme exhibits linear dependence, with respect to the unknown system's order, on the number of multiplications/additions and an O(Llog2L) dependence on sorting operations, where L is the length of the system/signal to be estimated. Numerical results are also given to validate the performance of the proposed method against the Least-Absolute Shrinkage and Selection Operator (LASSO) algorithm and two very recently developed adaptive sparse schemes that fuse arguments from the LMS/RLS adaptation mechanisms with those imposed by the lasso rational.

[1]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[2]  Heinz H. Bauschke,et al.  A Weak-to-Strong Convergence Principle for Fejé-Monotone Methods in Hilbert Spaces , 2001, Math. Oper. Res..

[3]  Sergios Theodoridis,et al.  Adaptive algorithm for sparse system identification using projections onto weighted ℓ1 balls , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[4]  Georgios B. Giannakis,et al.  Online Adaptive Estimation of Sparse Signals: Where RLS Meets the $\ell_1$ -Norm , 2010, IEEE Transactions on Signal Processing.

[5]  Yoram Singer,et al.  Efficient projections onto the l1-ball for learning in high dimensions , 2008, ICML '08.

[6]  Stephen J. Wright,et al.  Toeplitz-Structured Compressed Sensing Matrices , 2007, 2007 IEEE/SP 14th Workshop on Statistical Signal Processing.

[7]  Vahid Tarokh,et al.  Asymptotic Achievability of the CramÉr–Rao Bound for Noisy Compressive Sampling , 2009, IEEE Transactions on Signal Processing.

[8]  Boris Polyak,et al.  The method of projections for finding the common point of convex sets , 1967 .

[9]  R.G. Baraniuk,et al.  Compressive Sensing [Lecture Notes] , 2007, IEEE Signal Processing Magazine.

[10]  J. Hiriart-Urruty,et al.  Convex analysis and minimization algorithms , 1993 .

[11]  Michael P. Friedlander,et al.  Probing the Pareto Frontier for Basis Pursuit Solutions , 2008, SIAM J. Sci. Comput..

[12]  Isao Yamada,et al.  Asymptotic minimization of sequences of loss functions constrained by families of quasi-nonexpansive mappings and its application to online learning , 2010, ArXiv.

[13]  Alfred O. Hero,et al.  Sparse LMS for system identification , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[14]  Simon Haykin,et al.  Adaptive filter theory (2nd ed.) , 1991 .

[15]  Jos F. Sturm,et al.  A Matlab toolbox for optimization over symmetric cones , 1999 .

[16]  E.J. Candes Compressive Sampling , 2022 .

[17]  Sergios Theodoridis,et al.  Adaptive Learning in a World of Projections , 2011, IEEE Signal Processing Magazine.

[18]  Emmanuel J. Candès,et al.  Decoding by linear programming , 2005, IEEE Transactions on Information Theory.

[19]  Georgios B. Giannakis,et al.  RLS-weighted Lasso for adaptive estimation of sparse signals , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[20]  I. Yamada,et al.  The Adaptive Projected Subgradient Method over the Fixed Point Set of Strongly Attracting Nonexpansive Mappings , 2006 .

[21]  I. Daubechies,et al.  Accelerated Projected Gradient Method for Linear Inverse Problems with Sparsity Constraints , 2007, 0706.4297.

[22]  Robert D. Nowak,et al.  Compressed Channel Sensing: A New Approach to Estimating Sparse Multipath Channels , 2010, Proceedings of the IEEE.

[23]  Stephen P. Boyd,et al.  Enhancing Sparsity by Reweighted ℓ1 Minimization , 2007, 0711.1612.

[24]  Holger Rauhut,et al.  Circulant and Toeplitz matrices in compressed sensing , 2009, ArXiv.

[25]  Heinz H. Bauschke,et al.  On Projection Algorithms for Solving Convex Feasibility Problems , 1996, SIAM Rev..

[26]  Paulo Sergio Ramirez,et al.  Fundamentals of Adaptive Filtering , 2002 .

[27]  S. Haykin,et al.  Adaptive Filter Theory , 1986 .

[28]  I. Yamada,et al.  Adaptive Projected Subgradient Method for Asymptotic Minimization of Sequence of Nonnegative Convex Functions , 2005 .

[29]  Yongyi Yang,et al.  Vector Space Projections: A Numerical Approach to Signal and Image Processing, Neural Nets, and Optics , 1998 .

[30]  Isao Yamada,et al.  An efficient robust adaptive filtering algorithm based on parallel subgradient projection techniques , 2002, IEEE Trans. Signal Process..

[31]  Isao Yamada,et al.  A sparse adaptive filtering using time-varying soft-thresholding techniques , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[32]  H. Zou The Adaptive Lasso and Its Oracle Properties , 2006 .

[33]  Sergios Theodoridis,et al.  Adaptive Constrained Learning in Reproducing Kernel Hilbert Spaces: The Robust Beamforming Case , 2009, IEEE Transactions on Signal Processing.

[34]  P. L. Combettes The foundations of set theoretic estimation , 1993 .

[35]  Sergios Theodoridis,et al.  Online Kernel-Based Classification Using Adaptive Projection Algorithms , 2008, IEEE Transactions on Signal Processing.