Algorithms for nonnegative matrix factorization with the beta-divergence

This paper describes algorithms for nonnegative matrix factorization (NMF) with the beta-divergence (beta-NMF). The beta-divergence is a family of cost functions parametrized by a single shape parameter beta that takes the Euclidean distance, the Kullback-Leibler divergence and the Itakura-Saito divergence as special cases (beta = 2,1,0, respectively). The proposed algorithms are based on a surrogate auxiliary function (a local majorization of the criterion function). We first describe a majorization-minimization (MM) algorithm that leads to multiplicative updates, which differ from standard heuristic multiplicative updates by a beta-dependent power exponent. The monotonicity of the heuristic algorithm can however be proven for beta in (0,1) using the proposed auxiliary function. Then we introduce the concept of majorization-equalization (ME) algorithm which produces updates that move along constant level sets of the auxiliary function and lead to larger steps than MM. Simulations on synthetic and real data illustrate the faster convergence of the ME approach. The paper also describes how the proposed algorithms can be adapted to two common variants of NMF : penalized NMF (i.e., when a penalty function of the factors is added to the criterion function) and convex-NMF (when the dictionary is assumed to belong to a known subspace).

[1]  L. Lucy.  An iterative technique for the rectification of observed distributions , 1974 .

[2]  Victoria Stodden,et al.  When Does Non-Negative Matrix Factorization Give a Correct Decomposition into Parts? , 2003, NIPS.

[3]  Pablo Tamayo,et al.  Metagenes and molecular pattern discovery using matrix factorization , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[4]  Vincent N. LaRiccia,et al.  ON EM-LIKE ALGORITHMS FOR MINIMUM DISTANCE ESTIMATION , 1998 .

[5]  Cédric Richard,et al.  Split gradient method for nonnegative matrix factorization , 2010, 2010 18th European Signal Processing Conference.

[6]  H Lantéri,et al.  A general method to devise maximum-likelihood signal restoration multiplicative algorithms with non-negativity constraints , 2001, Signal Process..

[7]  D. Hunter,et al.  A Tutorial on MM Algorithms , 2004 .

[8]  Ali Taylan Cemgil,et al.  Bayesian Inference for Nonnegative Matrix Factorisation Models , 2009, Comput. Intell. Neurosci..

[9]  Chris H. Q. Ding,et al.  Convex and Semi-Nonnegative Matrix Factorizations , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  V. P. Pauca,et al.  Nonnegative matrix factorization for spectral data analysis , 2006 .

[11]  Derry Fitzgerald,et al.  On the use of the beta divergence for musical source separation , 2009 .

[12]  Guillaume Lemaitre,et al.  Real-time Polyphonic Music Transcription with Non-negative Matrix Factorization and Beta-divergence , 2010, ISMIR.

[13]  Ruslan Salakhutdinov,et al.  Adaptive Overrelaxed Bound Optimization Methods , 2003, ICML.

[14]  H. Kameoka,et al.  Convergence-guaranteed multiplicative algorithms for nonnegative matrix factorization with β-divergence , 2010, 2010 IEEE International Workshop on Machine Learning for Signal Processing.

[15]  Yuan Gao,et al.  Improving molecular cancer class discovery through sparse non-negative matrix factorization , 2005 .

[16]  D. Titterington.  On the Iterative Image Space Reconstruction Algorthm for ECT , 1987, IEEE Transactions on Medical Imaging.

[17]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[18]  Yu Cao,et al.  Cross Burg entropy maximization and its application to ringing suppression in image reconstruction , 1999, IEEE Trans. Image Process..

[19]  Raul Kompass,et al.  A Generalized Divergence Measure for Nonnegative Matrix Factorization , 2007, Neural Computation.

[20]  I. Dhillon,et al.  Fast Projection-Based Methods for the Least Squares Nonnegative Matrix Approximation Problem , 2008 .

[21]  M. Mørup,et al.  Multiplicative Updates for the Lasso , 2007, 2007 IEEE Workshop on Machine Learning for Signal Processing.

[22]  Derek Greene,et al.  Ensemble non-negative matrix factorization methods for clustering protein-protein interactions , 2008, Bioinform..

[23]  Inderjit S. Dhillon,et al.  Generalized Nonnegative Matrix Approximations with Bregman Divergences , 2005, NIPS.

[24]  Barak A. Pearlmutter,et al.  Convolutive Non-Negative Matrix Factorisation with a Sparseness Constraint , 2006 .

[25]  P. Smaragdis,et al.  Non-negative matrix factorization for polyphonic music transcription , 2003, 2003 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (IEEE Cat. No.03TH8684).

[26]  Chih-Jen Lin,et al.  On the Convergence of Multiplicative Update Algorithms for Nonnegative Matrix Factorization , 2007, IEEE Transactions on Neural Networks.

[27]  Nancy Bertin,et al.  Nonnegative Matrix Factorization with the Itakura-Saito Divergence: With Application to Music Analysis , 2009, Neural Computation.

[28]  Guillermo Sapiro,et al.  Online Learning for Matrix Factorization and Sparse Coding , 2009, J. Mach. Learn. Res..

[29]  Upendra Prasad.  Nonnegative matrix factorization: Analysis, algorithms and applications , 2009 .

[30]  Roland Badeau,et al.  NMF With Time–Frequency Activations to Model Nonstationary Audio Events , 2011, IEEE Transactions on Audio, Speech, and Language Processing.

[31]  Andrzej Cichocki,et al.  Non-Negative Matrix Factorization , 2020 .

[32]  Michael W. Berry,et al.  Algorithms and applications for approximate nonnegative matrix factorization , 2007, Comput. Stat. Data Anal..

[33]  Roland Badeau,et al.  A tempering approach for Itakura-Saito non-negative matrix factorization. With application to music transcription , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[34]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[35]  Chih-Jen Lin,et al.  Projected Gradient Methods for Nonnegative Matrix Factorization , 2007, Neural Computation.

[36]  M. E. Daube-Witherspoon,et al.  An iterative image space reconstruction algorithm suitable for volume ECT.IEEE Trans. , 1986 .

[37]  Andrzej Cichocki,et al.  Csiszár's Divergences for Non-negative Matrix Factorization: Family of New Algorithms , 2006, ICA.

[38]  William H. Richardson,et al.  Bayesian-Based Iterative Method of Image Restoration , 1972 .

[39]  Mark D. Plumbley,et al.  Theorems on Positive Data: On the Uniqueness of NMF , 2008, Comput. Intell. Neurosci..

[40]  Andrzej Cichocki,et al.  Families of Alpha- Beta- and Gamma- Divergences: Flexible and Robust Measures of Similarities , 2010, Entropy.

[41]  Paris Smaragdis,et al.  Missing data imputation for spectral audio signals , 2009, 2009 IEEE International Workshop on Machine Learning for Signal Processing.

[42]  Hirokazu Kameoka,et al.  Computational auditory induction by missing-data non-negative matrix factorization , 2008, SAPA@INTERSPEECH.

[43]  Emmanuel Vincent,et al.  Adaptive Harmonic Spectral Decomposition for Multiple Pitch Estimation , 2010, IEEE Transactions on Audio, Speech, and Language Processing.

[44]  A. R. De Pierro,et al.  On the relation between the ISRA and the EM algorithm for positron emission tomography , 1993, IEEE Trans. Medical Imaging.

[45]  Konstantinos Drakakis,et al.  Analysis of Financial Data Using , 2008 .

[46]  Yin Zhang,et al.  Accelerating the Lee-Seung Algorithm for Nonnegative Matrix Factorization , 2005 .

[47]  M. C. Jones,et al.  Robust and efficient estimation by minimising a density power divergence , 1998 .

[48]  Lars Kai Hansen,et al.  Tuning pruning in sparse non-negative matrix factorization , 2009, 2009 17th European Signal Processing Conference.

[49]  Mihoko Minami,et al.  Robust Blind Source Separation by Beta Divergence , 2002, Neural Computation.

[50]  Ali Taylan Cemgil,et al.  Nonnegative matrix factorizations as probabilistic inference in composite models , 2009, 2009 17th European Signal Processing Conference.

[51]  Roland Badeau,et al.  Supporting document for the paper "Stability analysis of multiplicative update algorithms and application to non-negative matrix factorization" , 2009 .