Reweighted nonnegative least-mean-square algorithm

Statistical inference subject to nonnegativity constraints is a frequently occurring problem in learning problems. The nonnegative least-mean-square (NNLMS) algorithm was derived to address such problems in an online way. This algorithm builds on a fixed-point iteration strategy driven by the Karush-Kuhn-Tucker conditions. It was shown to provide low variance estimates, but it however suffers from unbalanced convergence rates of these estimates. In this paper, we address this problem by introducing a variant of the NNLMS algorithm. We provide a theoretical analysis of its behavior in terms of transient learning curve, steady-state and tracking performance. We also introduce an extension of the algorithm for online sparse system identification. Monte-Carlo simulations are conducted to illustrate the performance of the algorithm and to validate the theoretical results. HighlightsWe proposed a variant of NN-LMS algorithm with balanced weight convergence rates.Accurate performance analysis is performed for a general nonstationarity model.The sparse system identification problem can be solved via the derived algorithm.

[1]  J. Minkoff,et al.  Comment on the "Unnecessary assumption of statistical independence between reference signal and filter weights in feedforward adaptive systems" , 2001, IEEE Trans. Signal Process..

[2]  Alfred O. Hero,et al.  Sparse LMS for system identification , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[3]  Jie Chen,et al.  Nonnegative Least-Mean-Square Algorithm , 2011, IEEE Transactions on Signal Processing.

[4]  Jie Chen,et al.  Variants of Non-Negative Least-Mean-Square Algorithm and Convergence Analysis , 2014, IEEE Transactions on Signal Processing.

[5]  Charles L. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[6]  José Carlos M. Bermudez,et al.  Stochastic analysis of the filtered-X LMS algorithm in systems with nonlinear secondary paths , 2002, IEEE Trans. Signal Process..

[7]  Jie Chen,et al.  Steady-State Performance of Non-Negative Least-Mean-Square Algorithm and Its Variants , 2014, IEEE Signal Processing Letters.

[8]  Rafal Zdunek,et al.  Improved Modeling of Highly Loaded UMTS Network with Nonnegative Constraints , 2006, 2006 IEEE 17th International Symposium on Personal, Indoor and Mobile Radio Communications.

[9]  Richard C. Puetter,et al.  DIGITAL IMAGE RECONSTRUCTION , 2005 .

[10]  Paul H. Calamai,et al.  Projected gradient methods for linearly constrained problems , 1987, Math. Program..

[11]  Tor Arne Johansen,et al.  Constrained and regularized system identification , 1997 .

[12]  R. Bro,et al.  A fast non‐negativity‐constrained least squares algorithm , 1997 .

[13]  Amnon Shashua,et al.  Nonnegative Sparse PCA , 2006, NIPS.

[14]  Ali H. Sayed,et al.  Adaptive Filters , 2008 .

[15]  Ali H. Sayed,et al.  Ability of adaptive filters to track carrier offsets and channel nonstationarities , 2002, IEEE Trans. Signal Process..

[16]  Daniel D. Lee,et al.  Statistical signal processing with nonnegativity constraints , 2003, INTERSPEECH.

[17]  Chih-Jen Lin,et al.  On the Convergence of Multiplicative Update Algorithms for Nonnegative Matrix Factorization , 2007, IEEE Transactions on Neural Networks.

[18]  Donghui Chen,et al.  Nonnegativity constraints in numerical analysis , 2009, The Birth of Numerical Analysis.

[19]  Maruti Kumar Mudunuru,et al.  A numerical framework for diffusion-controlled bimolecular-reactive systems to enforce maximum principles and the non-negative constraint , 2012, J. Comput. Phys..

[20]  Mário A. T. Figueiredo,et al.  Gradient Projection for Sparse Reconstruction: Application to Compressed Sensing and Other Inverse Problems , 2007, IEEE Journal of Selected Topics in Signal Processing.

[21]  Sergios Theodoridis,et al.  Adaptive Learning in a World of Projections , 2011, IEEE Signal Processing Magazine.

[22]  Daniel D. Lee,et al.  Bayesian regularization and nonnegative deconvolution for room impulse response estimation , 2006, IEEE Transactions on Signal Processing.

[23]  Johnathan M. Bardsley,et al.  THE STABILIZING PROPERTIES OF NONNEGATIVITY CONSTRAINTS IN LEAST-SQUARES IMAGE RECONSTRUCTION , 2007 .

[24]  Jordan L. Boyd-Graber,et al.  Maximum Likelihood , 2006 .

[25]  Yong Xiang,et al.  Nonnegative Blind Source Separation by Sparse Component Analysis Based on Determinant Measure , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[26]  Donald L. Duttweiler,et al.  Proportionate normalized least-mean-squares adaptation in echo cancelers , 2000, IEEE Trans. Speech Audio Process..

[27]  S. Haykin,et al.  Adaptive Filter Theory , 1986 .

[28]  Paul Honeine,et al.  Identification en ligne avec régularisation L1. Algorithme et analyse de convergence en environnement non-stationnaire , 2013 .

[29]  H Lantéri,et al.  A general method to devise maximum-likelihood signal restoration multiplicative algorithms with non-negativity constraints , 2001, Signal Process..

[30]  Yong Xiang,et al.  A Convex Geometry-Based Blind Source Separation Method for Separating Nonnegative Sources , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[31]  Alexandros G. Dimakis,et al.  Sparse Recovery of Nonnegative Signals With Minimal Expansion , 2011, IEEE Transactions on Signal Processing.

[32]  Jacek M. Zurada,et al.  Learning Understandable Neural Networks With Nonnegative Weight Constraints , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[33]  John F. Mustard,et al.  Spectral unmixing , 2002, IEEE Signal Process. Mag..

[34]  T. R. Gosnell,et al.  Digital Image Reconstruction: Deblurring and Denoising , 2005 .

[35]  Zhigang Luo,et al.  Online Nonnegative Matrix Factorization With Robust Stochastic Approximation , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[36]  Linda Kaufman,et al.  Maximum likelihood, least squares, and penalized least squares for PET , 1993, IEEE Trans. Medical Imaging.

[37]  Stephen P. Boyd,et al.  Enhancing Sparsity by Reweighted ℓ1 Minimization , 2007, 0711.1612.

[38]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[39]  Arie Yeredor,et al.  On the Role of Constraints in System Identification , 2006 .