Approximation Algorithms for Orthogonal Non-negative Matrix Factorization

In the non-negative matrix factorization (NMF) problem, the input is an m×n matrix M with non-negative entries and the goal is to factorize it as M ≈ AW . The m × k matrix A and the k×n matrix W are both constrained to have non-negative entries. This is in contrast to singular value decomposition, where the matrices A and W can have negative entries but must satisfy the orthogonality constraint: the columns of A are orthogonal and the rows of W are also orthogonal. The orthogonal non-negative matrix factorization (ONMF) problem imposes both the non-negativity and the orthogonality constraints, and previous work showed that it leads to better performances than NMF on many clustering tasks. We give the first constant-factor approximation algorithm for ONMF when one or both of A and W are subject to the orthogonality constraint. We also show an interesting connection to the correlation clustering problem on bipartite graphs. Our experiments on synthetic and real-world data show that our algorithm achieves similar or smaller errors compared to previous ONMF algorithms while ensuring perfect orthogonality (many previous algorithms do not satisfy the hard orthogonality constraint).

[1]  Chih-Jen Lin,et al.  Projected Gradient Methods for Nonnegative Matrix Factorization , 2007, Neural Computation.

[2]  Dan Feldman,et al.  A PTAS for k-means clustering based on weak coresets , 2007, SCG '07.

[3]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[4]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[5]  Ola Svensson,et al.  Better Guarantees for k-Means and Euclidean k-Median by Primal-Dual Algorithms , 2016, 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS).

[6]  Haesun Park,et al.  Fast Nonnegative Matrix Factorization: An Active-Set-Like Method and Comparisons , 2011, SIAM J. Sci. Comput..

[7]  Seungjin Choi,et al.  Orthogonal Nonnegative Matrix Factorization: Multiplicative Updates on Stiefel Manifolds , 2008, IDEAL.

[8]  Jorma Laaksonen,et al.  Multiplicative updates for non-negative projections , 2007, Neurocomputing.

[9]  Yi Yang,et al.  Discriminative Orthogonal Nonnegative matrix factorization with flexibility for data representation , 2014, Expert Syst. Appl..

[10]  P. Paatero,et al.  Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values† , 1994 .

[11]  Michael W. Berry,et al.  Text Mining Using Non-Negative Matrix Factorizations , 2004, SDM.

[12]  Karthik Devarajan,et al.  Nonnegative Matrix Factorization: An Analytical and Interpretive Tool in Computational Biology , 2008, PLoS Comput. Biol..

[13]  Chris H. Q. Ding,et al.  Orthogonal nonnegative matrix t-factorizations for clustering , 2006, KDD '06.

[14]  Annie Qu,et al.  Semi-Orthogonal Non-Negative Matrix Factorization , 2018, ArXiv.

[15]  David M. Mount,et al.  A local search approximation algorithm for k-means clustering , 2002, SCG '02.

[16]  Haibin Di,et al.  Weakly Supervised Seismic Structure Labeling via Orthogonal Non-Negative Matrix Factorization , 2017 .

[17]  Erkki Oja,et al.  Linear and Nonlinear Projective Nonnegative Matrix Factorization , 2010, IEEE Transactions on Neural Networks.

[18]  Xin Liu,et al.  Document clustering based on non-negative matrix factorization , 2003, SIGIR.

[19]  Yaoyao Qin,et al.  Community detection using nonnegative matrix factorization with orthogonal constraint , 2016, 2016 Eighth International Conference on Advanced Computational Intelligence (ICACI).

[20]  L. Mirsky.  SYMMETRIC GAUGE FUNCTIONS AND UNITARILY INVARIANT NORMS , 1960 .

[21]  Nir Ailon,et al.  Aggregating inconsistent information: Ranking and clustering , 2008 .

[22]  Seungjin Choi,et al.  Algorithms for orthogonal nonnegative matrix factorization , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[23]  C. Eckart,et al.  The approximation of one matrix by another of lower rank , 1936 .

[24]  Chris H. Q. Ding,et al.  Symmetric Nonnegative Matrix Factorization for Graph Clustering , 2012, SDM.

[25]  Andrzej Cichocki,et al.  Two Efficient Algorithms for Approximately Orthogonal Nonnegative Matrix Factorization , 2015, IEEE Signal Processing Letters.

[26]  Michael K. Ng,et al.  Orthogonal Nonnegative Matrix Factorization by Sparsity and Nuclear Norm Optimization , 2018, SIAM J. Matrix Anal. Appl..

[27]  Nicoletta Del Buono.  A Penalty Function for Computing Orthogonal Non-negative Matrix Factorizations , 2009, 2009 Ninth International Conference on Intelligent Systems Design and Applications.

[28]  Erkki Oja,et al.  Projective Nonnegative Matrix Factorization for Image Compression and Feature Extraction , 2005, SCIA.

[29]  Huifang Ma,et al.  Orthogonal Nonnegative Matrix Tri-factorization for Semi-supervised Document Co-clustering , 2010, PAKDD.

[30]  Yong Chen,et al.  Soft orthogonal non-negative matrix factorization with sparse representation: Static and dynamic , 2018, Neurocomputing.

[31]  Lee Sael,et al.  A mutation profile for top-k patient search exploiting Gene-Ontology and orthogonal non-negative matrix factorization , 2016, Bioinform..

[32]  Dimitris S. Papailiopoulos,et al.  Orthogonal NMF through Subspace Exploration , 2015, NIPS.

[33]  Lina Yao,et al.  Efficient Orthogonal Non-negative Matrix Factorization over Stiefel Manifold , 2016, CIKM.

[34]  D. Botstein,et al.  Singular value decomposition for genome-wide expression data processing and modeling. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[35]  Seungjin Choi,et al.  Nonnegative Matrix Factorization with Orthogonality Constraints , 2010, J. Comput. Sci. Eng..

[36]  Nicolas Gillis,et al.  ONP-MF: An Orthogonal Nonnegative Matrix Factorization Algorithm with Application to Clustering , 2013, ESANN.

[37]  Santosh S. Vempala,et al.  Latent semantic indexing: a probabilistic analysis , 1998, PODS '98.

[38]  Yuzuru Tanaka,et al.  A column-wise update algorithm for nonnegative matrix factorization in Bregman divergence with an orthogonal constraint , 2016, Machine Learning.

[39]  Anthony Wirth,et al.  Correlation Clustering , 2010, Encyclopedia of Machine Learning and Data Mining.

[40]  Sanjeev Arora,et al.  Computing a nonnegative matrix factorization -- provably , 2011, STOC '12.

[41]  Ping He,et al.  Low-rank nonnegative matrix factorization on Stiefel manifold , 2020, Inf. Sci..

[42]  Wenbo Li,et al.  Two fast vector-wise update algorithms for orthogonal nonnegative matrix factorization with sparsity constraint , 2020, J. Comput. Appl. Math..

[43]  Liu Bo,et al.  Collaborative Filtering Based on Orthogonal Non-negative Matrix Factorization , 2019, Journal of Physics: Conference Series.

[44]  Hyunsoo Kim,et al.  Sparse Non-negative Matrix Factorizations via Alternating Non-negativity-constrained Least Squares , 2006 .

[45]  Stan Z. Li,et al.  Learning spatially localized, parts-based representation , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[46]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[47]  Gang Chen,et al.  Collaborative Filtering Using Orthogonal Nonnegative Matrix Tri-factorization , 2007, Seventh IEEE International Conference on Data Mining Workshops (ICDMW 2007).

[48]  Pablo Tamayo,et al.  Metagenes and molecular pattern discovery using matrix factorization , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[49]  Tselil Schramm,et al.  Near Optimal LP Rounding Algorithm for CorrelationClustering on Complete and Complete k-partite Graphs , 2014, STOC.

[50]  Nicolas Gillis,et al.  Two algorithms for orthogonal nonnegative matrix factorization with application to clustering , 2012, Neurocomputing.

[51]  Stephen A. Vavasis,et al.  On the Complexity of Nonnegative Matrix Factorization , 2007, SIAM J. Optim..