The Why and How of Nonnegative Matrix Factorization

Nonnegative matrix factorization (NMF) has become a widely used tool for the analysis of high-dimensional data as it automatically extracts sparse and meaningful features from a set of nonnegative data vectors. We first illustrate this property of NMF on three applications, in image processing, text mining and hyperspectral imaging --this is the why. Then we address the problem of solving NMF, which is NP-hard in general. We review some standard NMF algorithms, and also present a recent subclass of NMF problems, referred to as near-separable NMF, that can be solved efficiently (that is, in polynomial time), even in the presence of noise --this is the how. Finally, we briefly describe some problems in mathematics and computer science closely related to NMF via the nonnegative rank.

[1]  Sophie Ahrens,et al.  Recommender Systems , 2012 .

[2]  Hyunsoo Kim,et al.  Sparse Non-negative Matrix Factorizations via Alternating Non-negativity-constrained Least Squares , 2006 .

[3]  Haesun Park,et al.  Fast bregman divergence NMF using taylor expansion and coordinate descent , 2012, KDD.

[4]  Malik Magdon-Ismail,et al.  On selecting a maximum volume sub-matrix of a matrix and related problems , 2009, Theor. Comput. Sci..

[5]  Joel A. Tropp,et al.  Factoring nonnegative matrices with linear programs , 2012, NIPS.

[6]  Chong-Yung Chi,et al.  Hyperspectral Data Geometry-Based Estimation of Number of Endmembers Using p-Norm-Based Pure Pixel Identification Algorithm , 2013, IEEE Transactions on Geoscience and Remote Sensing.

[7]  Samuel Fiorini,et al.  Combinatorial bounds on nonnegative rank and extended formulations , 2011, Discret. Math..

[8]  Paris Smaragdis,et al.  Static and Dynamic Source Separation Using Nonnegative Factorizations: A unified view , 2014, IEEE Signal Processing Magazine.

[9]  Nicolas Gillis,et al.  Accelerated Multiplicative Updates and Hierarchical ALS Algorithms for Nonnegative Matrix Factorization , 2011, Neural Computation.

[10]  Rasmus Bro,et al.  MULTI-WAY ANALYSIS IN THE FOOD INDUSTRY Models, Algorithms & Applications , 1998 .

[11]  Patrik O. Hoyer,et al.  Non-negative Matrix Factorization with Sparseness Constraints , 2004, J. Mach. Learn. Res..

[12]  Wady Naanaa,et al.  Blind source separation of positive and partially correlated data , 2005, Signal Process..

[13]  Jieping Ye,et al.  Sparse non-negative tensor factorization using columnwise coordinate descent , 2012, Pattern Recognit..

[14]  Éric Gaussier,et al.  Relation between PLSA and NMF and implications , 2005, SIGIR '05.

[15]  Hairong Qi,et al.  Endmember Extraction From Highly Mixed Data Using Minimum Volume Constrained Nonnegative Matrix Factorization , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[16]  Tamara G. Kolda,et al.  On Tensors, Sparsity, and Nonnegative Factorizations , 2011, SIAM J. Matrix Anal. Appl..

[17]  Rafal Zdunek.  Initialization of Nonnegative Matrix Factorization with Vertices of Convex Polytope , 2012, ICAISC.

[18]  Lixing Han,et al.  On the rate of convergence of the image space reconstruction algorithm , 2009 .

[19]  C. Févotte,et al.  Automatic Relevance Determination in Nonnegative Matrix Factorization , 2009 .

[20]  Nancy Bertin,et al.  Nonnegative Matrix Factorization with the Itakura-Saito Divergence: With Application to Music Analysis , 2009, Neural Computation.

[21]  Sabine Van Huffel,et al.  Hierarchical non‐negative matrix factorization (hNMF): a tissue pattern differentiation method for glioblastoma multiforme diagnosis using MRSI , 2013, NMR in biomedicine.

[22]  Norikazu Takahashi,et al.  Global convergence of modified multiplicative updates for nonnegative matrix factorization , 2013, Computational Optimization and Applications.

[23]  Venkatesh Saligrama,et al.  Topic Discovery through Data Dependent and Random Projections , 2013, ICML.

[24]  Antonio J. Plaza,et al.  Total Variation Spatial Regularization for Sparse Hyperspectral Unmixing , 2012, IEEE Transactions on Geoscience and Remote Sensing.

[25]  Upendra Prasad.  Nonnegative matrix factorization: Analysis, algorithms and applications , 2009 .

[26]  Malik Magdon-Ismail,et al.  Exponential Inapproximability of Selecting a Maximum Volume Sub-matrix , 2011, Algorithmica.

[27]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[28]  Luigi Grippo,et al.  On the convergence of the block nonlinear Gauss-Seidel method under convex constraints , 2000, Oper. Res. Lett..

[29]  Marian-Daniel Iordache,et al.  Greedy algorithms for pure pixels identification in hyperspectral unmixing: A multiple-measurement vector viewpoint , 2013, 21st European Signal Processing Conference (EUSIPCO 2013).

[30]  Andrzej Cichocki,et al.  Hierarchical ALS Algorithms for Nonnegative Matrix and 3D Tensor Factorization , 2007, ICA.

[31]  Ankur Moitra,et al.  An Almost Optimal Algorithm for Computing Nonnegative Rank , 2013, SIAM J. Comput..

[32]  Xiaojun Wu,et al.  Graph Regularized Nonnegative Matrix Factorization for Data Representation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Antonio J. Plaza,et al.  Hyperspectral Unmixing Overview: Geometrical, Statistical, and Sparse Regression-Based Approaches , 2012, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[34]  David B. Dunson,et al.  Probabilistic topic models , 2011, KDD '11 Tutorials.

[35]  Gérard Cornuéjols,et al.  Extended formulations in combinatorial optimization , 2010, 4OR.

[36]  Joydeep Ghosh,et al.  Under Consideration for Publication in Knowledge and Information Systems Generative Model-based Document Clustering: a Comparative Study , 2003 .

[37]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[38]  Chris H. Q. Ding,et al.  On the equivalence between Non-negative Matrix Factorization and Probabilistic Latent Semantic Indexing , 2008, Comput. Stat. Data Anal..

[39]  Karthik Devarajan,et al.  Nonnegative Matrix Factorization: An Analytical and Interpretive Tool in Computational Biology , 2008, PLoS Comput. Biol..

[40]  Chein-I Chang,et al.  Automatic spectral target recognition in hyperspectral imagery , 2003 .

[41]  Chih-Jen Lin,et al.  Projected Gradient Methods for Nonnegative Matrix Factorization , 2007, Neural Computation.

[42]  Fei Wang,et al.  Community discovery using nonnegative matrix factorization , 2011, Data Mining and Knowledge Discovery.

[43]  Nicolas Gillis,et al.  Robustness Analysis of Hottopixx, a Linear Programming Model for Factoring Nonnegative Matrices , 2012, SIAM J. Matrix Anal. Appl..

[44]  Guillermo Sapiro,et al.  See all by looking at a few: Sparse modeling for finding representative objects , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[45]  Erkki Oja,et al.  Linear and Nonlinear Projective Nonnegative Matrix Factorization , 2010, IEEE Transactions on Neural Networks.

[46]  Zhigang Luo,et al.  NeNMF: An Optimal Gradient Method for Nonnegative Matrix Factorization , 2012, IEEE Transactions on Signal Processing.

[47]  Seungjin Choi,et al.  Algorithms for orthogonal nonnegative matrix factorization , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[48]  Charles L. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[49]  Nicolas Gillis,et al.  Robust near-separable nonnegative matrix factorization using linear optimization , 2013, J. Mach. Learn. Res..

[50]  Andrzej Cichocki,et al.  Non-negative Matrix Factorization with Quasi-Newton Optimization , 2006, ICAISC.

[51]  Michael W. Berry,et al.  Algorithms and applications for approximate nonnegative matrix factorization , 2007, Comput. Stat. Data Anal..

[52]  Chong-Yung Chi,et al.  A Convex Analysis Framework for Blind Separation of Non-Negative Sources , 2008, IEEE Transactions on Signal Processing.

[53]  P. Paatero,et al.  Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values† , 1994 .

[54]  Tao Li,et al.  A Non-negative Matrix Tri-factorization Approach to Sentiment Classification with Lexical Prior Knowledge , 2009, ACL.

[55]  Nicolas Gillis,et al.  On the Geometric Interpretation of the Nonnegative Rank , 2010, 1009.0880.

[56]  José M. Bioucas-Dias,et al.  Vertex component analysis: a fast algorithm to unmix hyperspectral data , 2005, IEEE Transactions on Geoscience and Remote Sensing.

[57]  M. Yannakakis.  Expressing combinatorial optimization problems by linear programs , 1991, Symposium on the Theory of Computing.

[58]  S. Vavasis,et al.  M L ] 7 O ct 2 01 3 Fast and Robust Recursive Algorithms for Separable Nonnegative Matrix Factorization ∗ , 2013 .

[59]  Christos Boutsidis,et al.  SVD based initialization: A head start for nonnegative matrix factorization , 2008, Pattern Recognit..

[60]  Vikas Sindhwani,et al.  Rank Selection in Low-rank Matrix Approximations : A Study of Cross-Validation for NMFs , 2010 .

[61]  Michael Lindenbaum,et al.  Nonnegative Matrix Factorization with Earth Mover's Distance metric , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[62]  A. Lefèvre.  Dictionary learning methods for single-channel source separation , 2012 .

[63]  Haesun Park,et al.  Algorithms for nonnegative matrix and tensor factorizations: a unified view based on block coordinate descent framework , 2014, J. Glob. Optim..

[64]  Chong-Yung Chi,et al.  A Simplex Volume Maximization Framework for Hyperspectral Endmember Extraction , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[65]  M. E. Daube-Witherspoon,et al.  An iterative image space reconstruction algorithm suitable for volume ECT.IEEE Trans. , 1986 .

[66]  Chris H. Q. Ding,et al.  Convex and Semi-Nonnegative Matrix Factorizations , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[67]  Ying Chen,et al.  Clustering-based initialization for non-negative matrix factorization , 2008, Appl. Math. Comput..

[68]  Vikas Sindhwani,et al.  Recommender Systems , 2017, Encyclopedia of Machine Learning and Data Mining.

[69]  Nicolas Gillis,et al.  Successive Nonnegative Projection Algorithm for Robust Nonnegative Blind Source Separation , 2013, SIAM J. Imaging Sci..

[70]  Nikos D. Sidiropoulos,et al.  Non-Negative Matrix Factorization Revisited: Uniqueness and Algorithm for Symmetric Decomposition , 2014, IEEE Transactions on Signal Processing.

[71]  Inderjit S. Dhillon,et al.  Fast coordinate descent methods with variable selection for non-negative matrix factorization , 2011, KDD.

[72]  Gene H. Golub,et al.  Matrix computations (3rd ed.) , 1996 .

[73]  Sen Jia,et al.  Constrained Nonnegative Matrix Factorization for Hyperspectral Unmixing , 2009, IEEE Transactions on Geoscience and Remote Sensing.

[74]  Jean-Luc Starck,et al.  Sparse and Non-Negative BSS for Noisy Data , 2013, IEEE Transactions on Signal Processing.

[75]  Troy Lee,et al.  Lower Bounds in Communication Complexity , 2009, Found. Trends Theor. Comput. Sci..

[76]  Paul D. Gader,et al.  A Signal Processing Perspective on Hyperspectral Unmixing , 2014 .

[77]  Michael Möller,et al.  A Convex Model for Nonnegative Matrix Factorization and Dimensionality Reduction on Physical Space , 2011, IEEE Transactions on Image Processing.

[78]  Stephen A. Vavasis,et al.  On the Complexity of Nonnegative Matrix Factorization , 2007, SIAM J. Optim..

[79]  Mario Winter,et al.  N-FINDR: an algorithm for fast autonomous spectral end-member determination in hyperspectral data , 1999, Optics & Photonics.

[80]  M. C. U. Araújo,et al.  The successive projections algorithm for variable selection in spectroscopic multicomponent analysis , 2001 .

[81]  Rekha R. Thomas,et al.  Approximate cone factorizations and lifts of polytopes , 2015, Math. Program..

[82]  G. Golub,et al.  Linear least squares solutions by householder transformations , 1965 .

[83]  Jordi Vitrià,et al.  Non-negative Matrix Factorization for Face Recognition , 2002, CCIA.

[84]  Andrzej Cichocki,et al.  Nonnegative Matrix and Tensor Factorization T , 2007 .

[85]  Samuel Fiorini,et al.  Approximation Limits of Linear Programs (Beyond Hierarchies) , 2015, Math. Oper. Res..

[86]  Haesun Park,et al.  Fast Nonnegative Matrix Factorization: An Active-Set-Like Method and Comparisons , 2011, SIAM J. Sci. Comput..

[87]  Christos Boutsidis,et al.  An improved approximation algorithm for the column subset selection problem , 2008, SODA.

[88]  Nicolas Gillis,et al.  Sparse and unique nonnegative matrix factorization through data preprocessing , 2012, J. Mach. Learn. Res..

[89]  Sanjeev Arora,et al.  Computing a nonnegative matrix factorization -- provably , 2011, STOC '12.

[90]  Takeo Kanade,et al.  Robust L/sub 1/ norm factorization in the presence of outliers and missing data by alternative convex programming , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[91]  Chris H. Q. Ding,et al.  Orthogonal nonnegative matrix t-factorizations for clustering , 2006, KDD '06.

[92]  Hyunsoo Kim,et al.  Nonnegative Matrix Factorization Based on Alternating Nonnegativity Constrained Least Squares and Active Set Method , 2008, SIAM J. Matrix Anal. Appl..

[93]  Nicolas Gillis,et al.  Using underapproximations for sparse nonnegative matrix factorization , 2009, Pattern Recognit..

[94]  Emmanuel Vincent,et al.  Stability Analysis of Multiplicative Update Algorithms and Application to Nonnegative Matrix Factorization , 2010, IEEE Transactions on Neural Networks.

[95]  Michael W. Berry,et al.  Document clustering using nonnegative matrix factorization , 2006, Inf. Process. Manag..

[96]  V. Kaibel.  Extended Formulations in Combinatorial Optimization , 2011, 1104.1023.

[97]  Vikas Sindhwani,et al.  Near-separable Non-negative Matrix Factorization with ℓ1 and Bregman Loss Functions , 2013, SDM.

[98]  Paul Resnick,et al.  Recommender systems , 1997, CACM.

[99]  Chih-Jen Lin,et al.  On the Convergence of Multiplicative Update Algorithms for Nonnegative Matrix Factorization , 2007, IEEE Transactions on Neural Networks.

[100]  Fabio Rapallo,et al.  Probability matrices, non-negative rank, and parameterization of mixture models , 2009, 0911.0412.

[101]  Corrado Mencar,et al.  Subtractive clustering for seeding non-negative matrix factorizations , 2014, Inf. Sci..

[102]  C. Ding,et al.  On the Equivalence of Nonnegative Matrix Factorization and K-means - Spectral Clustering , 2005 .

[103]  Vikas Sindhwani,et al.  Fast Conical Hull Algorithms for Near-separable Non-negative Matrix Factorization , 2012, ICML.

[104]  José M. Bioucas-Dias,et al.  Estimation of signal subspace on hyperspectral data , 2005, SPIE Remote Sensing.

[105]  Michael I. Jordan,et al.  A Direct Formulation for Sparse Pca Using Semidefinite Programming , 2004, NIPS 2004.

[106]  Petros Drineas,et al.  CUR matrix decompositions for improved data analysis , 2009, Proceedings of the National Academy of Sciences.

[107]  Nicolas Gillis.  Nonnegative matrix factorization : complexity, algorithms and applications , 2011 .

[108]  C. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[109]  Stefan M. Wild,et al.  Improving non-negative matrix factorizations through structured initialization , 2004, Pattern Recognit..

[110]  Rong Ge,et al.  Provable Algorithms for Machine Learning Problems , 2013 .

[111]  Nicolas Gillis,et al.  Fast and Robust Recursive Algorithmsfor Separable Nonnegative Matrix Factorization , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[112]  Yu-Jin Zhang,et al.  FastNMF: highly efficient monotonic fixed-point nonnegative matrix factorization algorithm with good applicability , 2009, J. Electronic Imaging.

[113]  Andrzej Cichocki,et al.  Fast Local Algorithms for Large Scale Nonnegative Matrix and Tensor Factorizations , 2009, IEICE Trans. Fundam. Electron. Commun. Comput. Sci..

[114]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[115]  Nicolas Gillis,et al.  Semidefinite Programming Based Preconditioning for More Robust Near-Separable Nonnegative Matrix Factorization , 2013, SIAM J. Optim..

[116]  Sanjeev Arora,et al.  A Practical Algorithm for Topic Modeling with Provable Guarantees , 2012, ICML.