Efficient and Non-Convex Coordinate Descent for Symmetric Nonnegative Matrix Factorization

Given a symmetric nonnegative matrix A, symmetric nonnegative matrix factorization (symNMF) is the problem of finding a nonnegative matrix H, usually with much fewer columns than A, such that A ≈ HHT. SymNMF can be used for data analysis and in particular for various clustering tasks. Unlike standard NMF, which is traditionally solved by a series of quadratic (convex) subproblems, we propose to solve symNMF by directly solving the nonconvex problem, namely, minimize ∥A - HHT ∥2, which is a fourth-order nonconvex problem. In this paper, we propose simple and very efficient coordinate descent schemes, which solve a series of fourth-order univariate subproblems exactly. We also derive convergence guarantees for our methods and show that they perform favorably compared to recent state-of-the-art methods on synthetic and real-world datasets, especially on large and sparse input matrices.

[1]  Girolamo Cardano,et al.  Ars magna or The rules of algebra , 1993 .

[2]  Nicolas Gillis,et al.  Accelerated Multiplicative Updates and Hierarchical ALS Algorithms for Nonnegative Matrix Factorization , 2011, Neural Computation.

[3]  David J. Buttler,et al.  Encyclopedia of Data Warehousing and Mining Second Edition , 2008 .

[4]  Peter J. C. Dickinson,et al.  On the computational complexity of membership problems for the completely positive cone and its dual , 2014, Comput. Optim. Appl..

[5]  Stephen Roberts,et al.  Overlapping community detection using Bayesian non-negative matrix factorization. , 2011, Physical review. E, Statistical, nonlinear, and soft matter physics.

[6]  Nicolas Gillis,et al.  Solving the Maximum Clique Problem with Symmetric Rank-One Non-negative Matrix Approximation , 2017, J. Optim. Theory Appl..

[7]  Jing Hua,et al.  Non-negative matrix factorization for semi-supervised data clustering , 2008, Knowledge and Information Systems.

[8]  Nikos D. Sidiropoulos,et al.  Non-Negative Matrix Factorization Revisited: Uniqueness and Algorithm for Symmetric Decomposition , 2014, IEEE Transactions on Signal Processing.

[9]  Stephen J. Wright.  Coordinate descent algorithms , 2015, Mathematical Programming.

[10]  Haesun Park,et al.  Fast Nonnegative Matrix Factorization: An Active-Set-Like Method and Comparisons , 2011, SIAM J. Sci. Comput..

[11]  Inderjit S. Dhillon,et al.  Fast coordinate descent methods with variable selection for non-negative matrix factorization , 2011, KDD.

[12]  Philipp Birken,et al.  Numerical Linear Algebra , 2011, Encyclopedia of Parallel Computing.

[13]  Chris H. Q. Ding,et al.  Symmetric Nonnegative Matrix Factorization for Graph Clustering , 2012, SDM.

[14]  Yu-Jin Zhang,et al.  FastNMF: highly efficient monotonic fixed-point nonnegative matrix factorization algorithm with good applicability , 2009, J. Electronic Imaging.

[15]  Andrzej Cichocki,et al.  Fast Local Algorithms for Large Scale Nonnegative Matrix and Tensor Factorizations , 2009, IEICE Trans. Fundam. Electron. Commun. Comput. Sci..

[16]  A. Berman,et al.  Completely Positive Matrices , 2003 .

[17]  Nicolas Gillis.  Nonnegative matrix factorization : complexity, algorithms and applications , 2011 .

[18]  Joydeep Ghosh,et al.  Under Consideration for Publication in Knowledge and Information Systems Generative Model-based Document Clustering: a Comparative Study , 2003 .

[19]  Xiaohui Yan,et al.  Learning Topics in Short Texts by Non-negative Matrix Factorization on Term Correlation Matrix , 2013, SDM.

[20]  Haesun Park,et al.  SymNMF: nonnegative low-rank approximation of a similarity matrix for graph clustering , 2014, Journal of Global Optimization.

[21]  Dimitri P. Bertsekas.  Corrections for the book NONLINEAR PROGRAMMING: 2ND , 2005 .

[22]  Shuzhong Zhang,et al.  Maximum Block Improvement and Polynomial Optimization , 2012, SIAM J. Optim..

[23]  Stephen P. Boyd,et al.  Generalized Low Rank Models , 2014, Found. Trends Mach. Learn..

[24]  Erkki Oja,et al.  Quadratic nonnegative matrix factorization , 2012, Pattern Recognit..

[25]  Erkki Oja,et al.  Clustering by Nonnegative Matrix Factorization Using Graph Random Walk , 2012, NIPS.

[26]  R. Bro,et al.  Resolving the sign ambiguity in the singular value decomposition , 2008 .

[27]  Amnon Shashua,et al.  A unifying approach to hard and probabilistic clustering , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[28]  Shang-Hua Teng,et al.  Spectral sparsification of graphs: theory and algorithms , 2013, CACM.

[29]  Haesun Park,et al.  Toward Faster Nonnegative Matrix Factorization: A New Algorithm and Comparisons , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[30]  Samuel Burer,et al.  On the copositive representation of binary and continuous nonconvex quadratic programs , 2009, Math. Program..

[31]  Inderjit S. Dhillon,et al.  Coordinate Descent Methods for Symmetric Nonnegative Matrix Factorization , 2015, ArXiv.

[32]  Efstratios Gallopoulos,et al.  Computing symmetric nonnegative rank factorizations , 2012 .

[33]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[34]  Patrick Seemann,et al.  Matrix Factorization Techniques for Recommender Systems , 2014 .

[35]  Philip S. Yu,et al.  Relational clustering by symmetric convex coding , 2007, ICML '07.

[36]  Yehuda Koren,et al.  Matrix Factorization Techniques for Recommender Systems , 2009, Computer.

[37]  Zhaoshui He,et al.  Symmetric Nonnegative Matrix Factorization: Algorithms and Applications to Probabilistic Clustering , 2011, IEEE Transactions on Neural Networks.

[38]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.