Heuristics for exact nonnegative matrix factorization

The exact nonnegative matrix factorization (exact NMF) problem is the following: given an m-by-n nonnegative matrix X and a factorization rank r, find, if possible, an m-by-r nonnegative matrix W and an r-by-n nonnegative matrix H such that $$X = WH$$X=WH. In this paper, we propose two heuristics for exact NMF, one inspired from simulated annealing and the other from the greedy randomized adaptive search procedure. We show empirically that these two heuristics are able to compute exact nonnegative factorizations for several classes of nonnegative matrices (namely, linear Euclidean distance matrices, slack matrices, unique-disjointness matrices, and randomly generated matrices) and as such demonstrate their superiority over standard multi-start strategies. We also consider a hybridization between these two heuristics that allows us to combine the advantages of both methods. Finally, we discuss the use of these heuristics to gain insight on the behavior of the nonnegative rank, i.e., the minimum factorization rank such that an exact NMF exists. In particular, we disprove a conjecture on the nonnegative rank of a Kronecker product, propose a new upper bound on the extension complexity of generic n-gons and conjecture the exact value of (i) the extension complexity of regular n-gons and (ii) the nonnegative rank of a submatrix of the slack matrix of the correlation polytope.

[1]  Yaroslav Shitov.  Nonnegative rank depends on the field II , 2016 .

[2]  Nicolas Gillis,et al.  Accelerated Multiplicative Updates and Hierarchical ALS Algorithms for Nonnegative Matrix Factorization , 2011, Neural Computation.

[3]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[4]  Haesun Park,et al.  Algorithms for nonnegative matrix and tensor factorizations: a unified view based on block coordinate descent framework , 2014, J. Glob. Optim..

[5]  Ying Tan,et al.  Iterative improvement of the Multiplicative Update NMF algorithm using nature-inspired optimization , 2011, 2011 Seventh International Conference on Natural Computation.

[6]  Ankur Moitra.  An Almost Optimal Algorithm for Computing Nonnegative Rank , 2013, SODA.

[7]  Rekha R. Thomas,et al.  Worst-case results for positive semidefinite rank , 2013, Mathematical Programming.

[8]  Thomas Rothvoß,et al.  The matching polytope has exponential extension complexity , 2013, STOC.

[9]  Mihalis Yannakakis,et al.  Expressing combinatorial optimization problems by linear programs , 1991, STOC '88.

[10]  Yaroslav Shitov.  An upper bound for nonnegative rank , 2014, J. Comb. Theory, Ser. A.

[11]  Nicolas Gillis,et al.  The Why and How of Nonnegative Matrix Factorization , 2014, ArXiv.

[12]  Ying Tan,et al.  Swarm Intelligence for Non-Negative Matrix Factorization , 2011, Int. J. Swarm Intell. Res..

[13]  Julian Pfeifle,et al.  Polygons as Sections of Higher-Dimensional Polytopes , 2014, Electron. J. Comb..

[14]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[15]  Hartmut Klauck,et al.  Dagstuhl Report 13082: Communication Complexity, Linear Optimization, and lower bounds for the nonnegative rank of matrices , 2013, ArXiv.

[16]  Nicolas Gillis,et al.  Semidefinite Programming Based Preconditioning for More Robust Near-Separable Nonnegative Matrix Factorization , 2013, SIAM J. Optim..

[17]  Joel E. Cohen,et al.  Nonnegative ranks, decompositions, and factorizations of nonnegative matrices , 1993 .

[18]  Zvi Galil,et al.  Lower bounds on communication complexity , 1984, STOC '84.

[19]  Christopher W. Brown.  QEPCAD B: a program for computing with semi-algebraic sets using CADs , 2003, SIGS.

[20]  Samuel Fiorini,et al.  Combinatorial bounds on nonnegative rank and extended formulations , 2011, Discret. Math..

[21]  Stefan Weltge,et al.  Computing The Extension Complexities of All 4-Dimensional 0/1-Polytopes , 2014, 1406.4895.

[22]  Hamza Fawzi,et al.  Rational and real positive semidefinite rank can be different , 2014, Oper. Res. Lett..

[23]  Volker Kaibel,et al.  A Short Proof that the Extension Complexity of the Correlation Polytope Grows Exponentially , 2013, Discret. Comput. Geom..

[24]  Andrzej Cichocki,et al.  Hierarchical ALS Algorithms for Nonnegative Matrix and 3D Tensor Factorization , 2007, ICA.

[25]  Andrzej Cichocki,et al.  Fast Local Algorithms for Large Scale Nonnegative Matrix and Tensor Factorizations , 2009, IEICE Trans. Fundam. Electron. Commun. Comput. Sci..

[26]  Fabio Rapallo,et al.  Perturbation of Matrices and Nonnegative Rank with a View toward Statistical Models , 2010, SIAM J. Matrix Anal. Appl..

[27]  A. Berman.  Rank Factorization of Nonnegative Matrices , 1973 .

[28]  M. Pirlot.  General local search methods , 1996 .

[29]  LeRoy B. Beasley,et al.  Real rank versus nonnegative rank , 2009 .

[30]  Yaroslav Shitov.  Sublinear extensions of polygons , 2014 .

[31]  Rafal Zdunek.  Initialization of Nonnegative Matrix Factorization with Vertices of Convex Polytope , 2012, ICAISC.

[32]  Rekha R. Thomas,et al.  Positive semidefinite rank , 2014, Math. Program..

[33]  Andrzej Cichocki,et al.  Nonnegative Matrix and Tensor Factorization T , 2007 .

[34]  Stephen A. Vavasis,et al.  On the Complexity of Nonnegative Matrix Factorization , 2007, SIAM J. Optim..

[35]  Rekha R. Thomas,et al.  Lifts of Convex Sets and Cone Factorizations , 2011, Math. Oper. Res..

[36]  Christos Boutsidis,et al.  SVD based initialization: A head start for nonnegative matrix factorization , 2008, Pattern Recognit..

[37]  Nicolas Gillis,et al.  On the Geometric Interpretation of the Nonnegative Rank , 2010, 1009.0880.

[38]  Arkadi Nemirovski,et al.  On Polyhedral Approximations of the Second-Order Cone , 2001, Math. Oper. Res..

[39]  Fabio Rapallo,et al.  Probability matrices, non-negative rank, and parameterization of mixture models , 2009, 0911.0412.

[40]  Nicolas Gillis,et al.  Using underapproximations for sparse nonnegative matrix factorization , 2009, Pattern Recognit..

[41]  V. Kaibel.  Extended Formulations in Combinatorial Optimization , 2011, 1104.1023.

[42]  Pavel Hrubes,et al.  On the nonnegative rank of distance matrices , 2012, Inf. Process. Lett..

[43]  Nicolas Gillis,et al.  On the Linear Extension Complexity of Regular n-gons , 2015, ArXiv.

[44]  Hans Raj Tiwary,et al.  Extended Formulations for Polygons , 2011, Discret. Comput. Geom..

[45]  Haesun Park,et al.  Fast Nonnegative Matrix Factorization: An Active-Set-Like Method and Comparisons , 2011, SIAM J. Sci. Comput..

[46]  Nicolas Gillis,et al.  Sparse and unique nonnegative matrix factorization through data preprocessing , 2012, J. Mach. Learn. Res..

[47]  Sanjeev Arora,et al.  Computing a nonnegative matrix factorization -- provably , 2011, STOC '12.

[48]  Hans Raj Tiwary,et al.  Exponential Lower Bounds for Polytopes in Combinatorial Optimization , 2011, J. ACM.

[49]  Michel X. Goemans,et al.  Smallest compact formulation for the permutahedron , 2015, Math. Program..

[50]  L. B. Thomas.  Rank Factorization of Nonnegative Matrices (A. Berman) , 1974 .

[51]  Ying Tan,et al.  Using Population Based Algorithms for Initializing Nonnegative Matrix Factorization , 2011, ICSI.

[52]  Norikazu Takahashi,et al.  Global convergence of modified multiplicative updates for nonnegative matrix factorization , 2013, Computational Optimization and Applications.