Putting nonnegative matrix factorization to the test: a tutorial derivation of pertinent cramer—rao bounds and performance benchmarking

Nonnegative matrix factorization (NMF) is a useful tool in a broad range of applications, from signal separation to computer vision and machine learning. NMF is a hard (NP-hard) computational problem for which various approximate solutions have been developed over the years. Given the widespread interest in NMF and its applications, it is perhaps surprising that the pertinent Cramer-Rao lower bound (CRLB) on the accuracy of the nonnegative latent factor estimates has not been worked out in the literature. In hindsight, one reason may be that the required computations are more subtle than usual: the problem involves constraints and ambiguities that must be dealt with, and the Fisher information matrix is always singular. We provide a concise tutorial derivation of the CRLB for both symmetric NMF and asymmetric NMF, using the latest CRLB tools, which should be of broad interest for analogous derivations in related factor analysis problems. We illustrate the behavior of these bounds with respect to model parameters and put some of the best NMF algorithms to the test against one another and the CRLB. The results help illuminate what can be expected from the current state of art in NMF algorithms, and they are reassuring in that the gap to optimality is small in relatively sparse and low rank scenarios.

[1]  Ananthram Swami,et al.  Cramer-Rao bounds for deterministic signals in additive and multiplicative noise , 1996, Signal Process..

[2]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[3]  Haesun Park,et al.  Fast Nonnegative Matrix Factorization: An Active-Set-Like Method and Comparisons , 2011, SIAM J. Sci. Comput..

[4]  Chih-Jen Lin,et al.  Projected Gradient Methods for Nonnegative Matrix Factorization , 2007, Neural Computation.

[5]  Moody T. Chu,et al.  Low-Dimensional Polytope Approximation and Its Applications to Nonnegative Matrix Factorization , 2008, SIAM J. Sci. Comput..

[6]  Nicolas Gillis,et al.  Sparse and unique nonnegative matrix factorization through data preprocessing , 2012, J. Mach. Learn. Res..

[7]  Zbynek Koldovský,et al.  Optimal pairing of signal components separated by blind techniques , 2004, IEEE Signal Processing Letters.

[8]  Harold W. Kuhn,et al.  The Hungarian method for the assignment problem , 1955, 50 Years of Integer Programming.

[9]  Rafal Zdunek.  Initialization of Nonnegative Matrix Factorization with Vertices of Convex Polytope , 2012, ICAISC.

[10]  Zhaoshui He,et al.  Symmetric Nonnegative Matrix Factorization: Algorithms and Applications to Probabilistic Clustering , 2011, IEEE Transactions on Neural Networks.

[11]  Christoph Schnörr,et al.  Learning Sparse Representations by Non-Negative Matrix Factorization and Sequential Cone Programming , 2006, J. Mach. Learn. Res..

[12]  James Curry,et al.  Non-negative matrix factorization: Ill-posedness and a geometric algorithm , 2009, Pattern Recognit..

[13]  Victoria Stodden,et al.  When Does Non-Negative Matrix Factorization Give a Correct Decomposition into Parts? , 2003, NIPS.

[14]  Alfred O. Hero,et al.  Lower bounds for parametric estimation with constraints , 1990, IEEE Trans. Inf. Theory.

[15]  T. Markham,et al.  The Moore-Penrose inverse of a sum of matrices , 1977, Journal of the Australian Mathematical Society.

[16]  Nikos D. Sidiropoulos,et al.  Non-Negative Matrix Factorization Revisited: Uniqueness and Algorithm for Symmetric Decomposition , 2014, IEEE Transactions on Signal Processing.

[17]  Michael W. Berry,et al.  Algorithms and applications for approximate nonnegative matrix factorization , 2007, Comput. Stat. Data Anal..

[18]  P. Paatero,et al.  Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values† , 1994 .

[19]  Stephen A. Vavasis,et al.  On the Complexity of Nonnegative Matrix Factorization , 2007, SIAM J. Optim..

[20]  Peter J. C. Dickinson,et al.  On the computational complexity of membership problems for the completely positive cone and its dual , 2014, Comput. Optim. Appl..

[21]  Erkki Oja,et al.  Quadratic nonnegative matrix factorization , 2012, Pattern Recognit..

[22]  Andrzej Cichocki,et al.  Nonnegative matrix factorization with constrained second-order optimization , 2007, Signal Process..

[23]  José M. Bioucas-Dias,et al.  Minimum Volume Simplex Analysis: A Fast Algorithm to Unmix Hyperspectral Data , 2008, IGARSS 2008 - 2008 IEEE International Geoscience and Remote Sensing Symposium.

[24]  Yonina C. Eldar,et al.  On the Constrained CramÉr–Rao Bound With a Singular Fisher Information Matrix , 2009, IEEE Signal Processing Letters.

[25]  B. C. Ng,et al.  On the Cramer-Rao bound under parametric constraints , 1998, IEEE Signal Processing Letters.

[26]  R. Plemmons,et al.  Optimality, computation, and interpretation of nonnegative matrix factorizations , 2004 .

[27]  Andrzej Cichocki,et al.  Nonnegative Matrix and Tensor Factorization T , 2007 .

[28]  Zhaoshui He,et al.  Extended SMART Algorithms for Non-negative Matrix Factorization , 2006, ICAISC.

[29]  Christian Jutten,et al.  Simplicial Cone Shrinking Algorithm for Unmixing Nonnegative Sources , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[30]  Ching-hsiang Hung,et al.  The Moore-Penrose inverse of a partitioned matrix ? , 1975 .

[31]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[32]  Thomas L. Marzetta,et al.  Parameter estimation problems with singular information matrices , 2001, IEEE Trans. Signal Process..

[33]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[34]  Chris H. Q. Ding,et al.  On the Equivalence of Nonnegative Matrix Factorization and Spectral Clustering , 2005, SDM.

[35]  Kaare Brandt Petersen,et al.  The Matrix Cookbook , 2006 .

[36]  Mark D. Plumbley,et al.  Theorems on Positive Data: On the Uniqueness of NMF , 2008, Comput. Intell. Neurosci..

[37]  Andrzej Cichocki,et al.  Fast Local Algorithms for Large Scale Nonnegative Matrix and Tensor Factorizations , 2009, IEICE Trans. Fundam. Electron. Commun. Comput. Sci..

[38]  Hyunsoo Kim,et al.  Nonnegative Matrix Factorization Based on Alternating Nonnegativity Constrained Least Squares and Active Set Method , 2008, SIAM J. Matrix Anal. Appl..