Sparsity-Aware Adaptive Learning: A Set Theoretic Estimation Approach

Abstract This paper reviews recent advances on online/adaptive sparsity-promoting algorithms. The emphasis is on on a recent family of schemes, which build upon convex analytic tools. The benefits of this algorithmic family is that it can easily deal with the existence of a set of convex constraints and also to bypass the need of differentiability of cost functions. It can thus deal well with notions related to robustness and their associated costs. Extensions to constraints, which are realized via mappings whose fixed point set are non-convex, are also discussed. The case of learning in a distributed fashion is also discussed.

[1]  Alfred O. Hero,et al.  Sparse LMS for system identification , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[2]  Isao Yamada,et al.  The Adaptive Projected Subgradient Method Constrained by Families of Quasi-nonexpansive Mappings and Its Application to Online Learning , 2010, SIAM J. Optim..

[3]  Sergios Theodoridis,et al.  Generalized thresholding sparsity-aware algorithm for low complexity online learning , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[4]  Sergios Theodoridis,et al.  Thresholding-based online algorithms of complexity comparable to sparse LMS methods , 2013, 2013 IEEE International Symposium on Circuits and Systems (ISCAS2013).

[5]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[6]  P. Diniz,et al.  Set-membership affine projection algorithm , 2001, IEEE Signal Processing Letters.

[7]  Jacob Benesty,et al.  An improved proportionate NLMS algorithm based on the l0 norm , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[8]  Heinz H. Bauschke,et al.  On Projection Algorithms for Solving Convex Feasibility Problems , 1996, SIAM Rev..

[9]  J. Friedman,et al.  [A Statistical View of Some Chemometrics Regression Tools]: Response , 1993 .

[10]  I. Yamada,et al.  Adaptive Projected Subgradient Method for Asymptotic Minimization of Sequence of Nonnegative Convex Functions , 2005 .

[11]  Sergios Theodoridis,et al.  Adaptive Robust Distributed Learning in Diffusion Sensor Networks , 2011, IEEE Transactions on Signal Processing.

[12]  D. Youla,et al.  Image Restoration by the Method of Convex Projections: Part 1ߞTheory , 1982, IEEE Transactions on Medical Imaging.

[13]  Sergios Theodoridis,et al.  Sparsity-Aware Learning and Compressed Sensing: An Overview , 2012, ArXiv.

[14]  Sergios Theodoridis,et al.  Stochastic Analysis of Hyperslab-Based Adaptive Projected Subgradient Method Under Bounded Noise , 2013, IEEE Signal Processing Letters.

[15]  Donald L. Duttweiler,et al.  Proportionate normalized least-mean-squares adaptation in echo cancelers , 2000, IEEE Trans. Speech Audio Process..

[16]  Jian Wang,et al.  Performance Analysis of $l_0$ Norm Constraint Least Mean Square Algorithm , 2012, IEEE Transactions on Signal Processing.

[17]  Minh N. Do,et al.  A Theory for Sampling Signals from a Union of Subspaces , 2022 .

[18]  Sergios Theodoridis,et al.  A Sparsity Promoting Adaptive Algorithm for Distributed Learning , 2012, IEEE Transactions on Signal Processing.

[19]  Cun-Hui Zhang.  Nearly unbiased variable selection under minimax concave penalty , 2010, 1002.4734.

[20]  Simon Foucart,et al.  Hard Thresholding Pursuit: An Algorithm for Compressive Sensing , 2011, SIAM J. Numer. Anal..

[21]  Yuantao Gu,et al.  $l_{0}$ Norm Constraint LMS Algorithm for Sparse System Identification , 2009, IEEE Signal Processing Letters.

[22]  Jianqing Fan,et al.  Regularization of Wavelet Approximations , 2001 .

[23]  Ali H. Sayed,et al.  Diffusion LMS Strategies for Distributed Estimation , 2010, IEEE Transactions on Signal Processing.

[24]  Isao Yamada,et al.  A sparse adaptive filtering using time-varying soft-thresholding techniques , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[25]  I. Yamada,et al.  The Adaptive Projected Subgradient Method over the Fixed Point Set of Strongly Attracting Nonexpansive Mappings , 2006 .

[26]  Mike E. Davies,et al.  Normalized Iterative Hard Thresholding: Guaranteed Stability and Performance , 2010, IEEE Journal of Selected Topics in Signal Processing.

[27]  Thomas Blumensath,et al.  Accelerated iterative hard thresholding , 2012, Signal Process..

[28]  Georgios B. Giannakis,et al.  Online Adaptive Estimation of Sparse Signals: Where RLS Meets the $\ell_1$ -Norm , 2010, IEEE Transactions on Signal Processing.

[29]  Yoram Singer,et al.  Efficient projections onto the l1-ball for learning in high dimensions , 2008, ICML '08.

[30]  David L Donoho,et al.  Compressed sensing , 2006, IEEE Transactions on Information Theory.

[31]  J. Friedman,et al.  A Statistical View of Some Chemometrics Regression Tools , 1993 .

[32]  Akihiko Sugiyama,et al.  A generalized proportionate variable step-size algorithm for fast changing acoustic environments , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[33]  P. L. Combettes,et al.  Foundation of set theoretic estimation , 1993 .

[34]  M. Chitre,et al.  New Sparse Adaptive Algorithms Based on the Natural Gradient and the ${L}_{0}$ -Norm , 2013, IEEE Journal of Oceanic Engineering.

[35]  Boris Polyak,et al.  The method of projections for finding the common point of convex sets , 1967 .

[36]  Yonina C. Eldar,et al.  Robust Recovery of Signals From a Structured Union of Subspaces , 2008, IEEE Transactions on Information Theory.

[37]  Vahid Tarokh,et al.  An Adaptive Greedy Algorithm With Application to Nonlinear Communications , 2010, IEEE Transactions on Signal Processing.

[38]  P. L. Combettes.  The foundations of set theoretic estimation , 1993 .

[39]  Ali H. Sayed,et al.  Diffusion Least-Mean Squares Over Adaptive Networks: Formulation and Performance Analysis , 2008, IEEE Transactions on Signal Processing.

[40]  Sergios Theodoridis,et al.  Generalized Thresholding and Online Sparsity-Aware Learning in a Union of Subspaces , 2011, IEEE Transactions on Signal Processing.

[41]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[42]  I. Daubechies,et al.  Accelerated Projected Gradient Method for Linear Inverse Problems with Sparsity Constraints , 2007, 0706.4297.

[43]  T. Hastie,et al.  SparseNet: Coordinate Descent With Nonconvex Penalties , 2011, Journal of the American Statistical Association.

[44]  Stephen P. Boyd,et al.  Enhancing Sparsity by Reweighted ℓ1 Minimization , 2007, 0711.1612.

[45]  Peng Shi,et al.  Convergence analysis of sparse LMS algorithms with l1-norm penalty based on white input signal , 2010, Signal Process..

[46]  W. A. Kirk,et al.  Topics in Metric Fixed Point Theory , 1990 .

[47]  I. Yamada,et al.  Low complexity projection-based adaptive algorithm for sparse system identification and signal reconstruction , 2010, 2010 Conference Record of the Forty Fourth Asilomar Conference on Signals, Systems and Computers.

[48]  Jacob Benesty,et al.  An Efficient Proportionate Affine Projection Algorithm for Echo Cancellation , 2010, IEEE Signal Processing Letters.

[49]  Hong-Ye Gao,et al.  Wavelet Shrinkage Denoising Using the Non-Negative Garrote , 1998 .

[50]  Vahid Tarokh,et al.  Asymptotic Achievability of the CramÉr–Rao Bound for Noisy Compressive Sampling , 2009, IEEE Transactions on Signal Processing.

[51]  Sergios Theodoridis,et al.  Online Sparse System Identification and Signal Reconstruction Using Projections Onto Weighted $\ell_{1}$ Balls , 2010, IEEE Transactions on Signal Processing.

[52]  Yongyi Yang,et al.  Vector Space Projections: A Numerical Approach to Signal and Image Processing, Neural Nets, and Optics , 1998 .

[53]  Ali H. Sayed,et al.  Fundamentals Of Adaptive Filtering , 2003 .

[54]  Sergios Theodoridis,et al.  Adaptive Learning in a World of Projections , 2011, IEEE Signal Processing Magazine.

[55]  Georgios B. Giannakis,et al.  RLS-weighted Lasso for adaptive estimation of sparse signals , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[56]  H. Zou.  The Adaptive Lasso and Its Oracle Properties , 2006 .

[57]  Vahid Tarokh,et al.  SPARLS: The Sparse RLS Algorithm , 2010, IEEE Transactions on Signal Processing.