Non-stationary analysis of the convergence of the Non-Negative Least-Mean-Square algorithm

Non-negativity is a widely used constraint in parameter estimation procedures due to physical characteristics of systems under investigation. In this paper, we consider an LMS-type algorithm for system identification subject to non-negativity constraints, called Non-Negative Least-Mean-Square algorithm, and its normalized variant. An important contribution of this paper is that we study the stochastic behavior of these algorithms in a non-stationary environment, where the unconstrained solution is characterized by a time-variant mean and is affected by random perturbations. Convergence analysis of these algorithms in a stationary environment can be viewed as a particular case of the convergence model derived in this paper. Simulation results are presented to illustrate the performance of the algorithm and the accuracy of the derived models.

[1]  Guillermo Sapiro,et al.  Online Learning for Matrix Factorization and Sparse Coding , 2009, J. Mach. Learn. Res..

[2]  V. Umapathi Reddy,et al.  Fixed point error analysis of the normalized ladder algorithm , 1982, ICASSP.

[3]  Ali H. Sayed,et al.  Adaptive Filters , 2008 .

[4]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[5]  H Lantéri,et al.  A general method to devise maximum-likelihood signal restoration multiplicative algorithms with non-negativity constraints , 2001, Signal Process..

[6]  Mark D. Plumbley.  Algorithms for nonnegative independent component analysis , 2003, IEEE Trans. Neural Networks.

[7]  Charles L. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[8]  Sérgio J. M. de Almeida,et al.  A statistical analysis of the affine projection algorithm for unity step size and autoregressive inputs , 2005, IEEE Transactions on Circuits and Systems I: Regular Papers.

[9]  D. Brie,et al.  Separation of Non-Negative Mixture of Non-Negative Sources Using a Bayesian Approach and MCMC Sampling , 2006, IEEE Transactions on Signal Processing.

[10]  J. Borwein,et al.  Two-Point Step Size Gradient Methods , 1988 .

[11]  Sergios Theodoridis,et al.  Adaptive Learning in a World of Projections , 2011, IEEE Signal Processing Magazine.

[12]  Michael W. Berry,et al.  Algorithms and applications for approximate nonnegative matrix factorization , 2007, Comput. Stat. Data Anal..

[13]  M. Bertero,et al.  Nonnegative least-squares image deblurring: improved gradient projection approaches , 2010 .

[14]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[15]  Jie Chen,et al.  Non-negative least-mean-square algorithm , 2011 .

[16]  David Wessel,et al.  Realtime Multiple-pitch and Multiple-instrument Recognition For Music Signals using Sparse Non-negative Constraints , 2007 .

[17]  Chih-Jen Lin,et al.  Projected Gradient Methods for Nonnegative Matrix Factorization , 2007, Neural Computation.

[18]  Paul H. Calamai,et al.  Projected gradient methods for linearly constrained problems , 1987, Math. Program..

[19]  J. B. Rosen.  The Gradient Projection Method for Nonlinear Programming. Part I. Linear Constraints , 1960 .

[20]  R. Bro,et al.  A fast non‐negativity‐constrained least squares algorithm , 1997 .

[21]  Andrzej Cichocki,et al.  Nonnegative Matrix and Tensor Factorization T , 2007 .

[22]  Daniel D. Lee,et al.  Bayesian regularization and nonnegative deconvolution for room impulse response estimation , 2006, IEEE Transactions on Signal Processing.

[23]  Ali H. Sayed,et al.  Diffusion Adaptation Over Networks Under Imperfect Information Exchange and Non-Stationary Data , 2011, IEEE Transactions on Signal Processing.

[24]  Charles L. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[25]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[26]  Jie Chen,et al.  Nonnegative Least-Mean-Square Algorithm , 2011, IEEE Transactions on Signal Processing.

[27]  Chih-Jen Lin,et al.  On the Convergence of Multiplicative Update Algorithms for Nonnegative Matrix Factorization , 2007, IEEE Transactions on Neural Networks.

[28]  LinYuanqing.  Bayesian regularization and nonnegative deconvolution for room impulse response estimation , 2006 .