A multilevel approach for nonnegative matrix factorization

Nonnegative matrix factorization (NMF), the problem of approximating a nonnegative matrix with the product of two low-rank nonnegative matrices, has been shown to be useful in many applications, such as text mining, image processing, and computational biology. In this paper, we explain how algorithms for NMF can be embedded into the framework of multilevel methods in order to accelerate their initial convergence. This technique can be applied in situations where data admit a good approximate representation in a lower dimensional space through linear transformations preserving nonnegativity. Several simple multilevel strategies are described and are experimentally shown to speed up significantly three popular NMF algorithms (alternating nonnegative least squares, multiplicative updates and hierarchical alternating least squares) on several standard image datasets.

[1]  Y. Nesterov,et al.  Double smoothing technique for infinite-dimensional optimization problems with applications to optimal control , 2010 .

[2]  Thierry Bréchet,et al.  Technological greening, eco-efficiency and no-regret strategy , 2010 .

[3]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[4]  Christos Boutsidis,et al.  SVD based initialization: A head start for nonnegative matrix factorization , 2008, Pattern Recognit..

[5]  M. E. Daube-Witherspoon,et al.  An iterative image space reconstruction algorithm suitable for volume ECT.IEEE Trans. , 1986 .

[6]  Axel Gautier,et al.  Universal Service Financing in Competitive Postal Markets: One Size Does Not Fit All , 2011 .

[7]  M. V. Van Benthem,et al.  Fast algorithm for the solution of large‐scale non‐negativity‐constrained least squares problems , 2004 .

[8]  Laurence A. Wolsey,et al.  Integration of AI and OR Techniques in Constraint Programming for Combinatorial Optimization Problems, 4th International Conference, CPAIOR 2007, Brussels, Belgium, May 23-26, 2007, Proceedings , 2007, CPAIOR.

[9]  Yurii Nesterov,et al.  Solving Infinite-dimensional Optimization Problems by Polynomial Approximation , 2010 .

[10]  KlingenbergBradley,et al.  Non-negative matrix factorization , 2009 .

[11]  Tanguy Isaac,et al.  When frictions favour information revelation , 2010 .

[12]  Haesun Park,et al.  Fast Nonnegative Matrix Factorization: An Active-Set-Like Method and Comparisons , 2011, SIAM J. Sci. Comput..

[13]  Andrzej Cichocki,et al.  Non-negative Matrix Factorization with Quasi-Newton Optimization , 2006, ICAISC.

[14]  Michael W. Berry,et al.  Algorithms and applications for approximate nonnegative matrix factorization , 2007, Comput. Stat. Data Anal..

[15]  P. Paatero,et al.  Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values† , 1994 .

[16]  James Renegar,et al.  A mathematical view of interior-point methods in convex optimization , 2001, MPS-SIAM series on optimization.

[17]  Elena Molis,et al.  The stability of the roommate problem revisited , 2010 .

[18]  Qiang Zhang,et al.  Tensor methods for hyperspectral data analysis: a space object material identification study. , 2008, Journal of the Optical Society of America. A, Optics, image science, and vision.

[19]  Haesun Park,et al.  Toward Faster Nonnegative Matrix Factorization: A New Algorithm and Comparisons , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[20]  Nicolas Gillis.  Nonnegative matrix factorization : complexity, algorithms and applications , 2011 .

[21]  Yves Smeers,et al.  Stochastic Equilibrium Models for Generation Capacity Expansion , 2010 .

[22]  Andrea Silvestrini,et al.  Aggregation of exponential smoothing processes with an application to portfolio risk evaluation , 2013 .

[23]  Rüdiger Stephan,et al.  An extension of disjunctive programming and its impact for compact tree formulations , 2010, 1007.1136.

[24]  E. Schokkaert,et al.  Equivalent income and the economic evaluation of health care , 2010 .

[25]  S. Amari,et al.  Nonnegative Matrix and Tensor Factorization [Lecture Notes] , 2008, IEEE Signal Processing Magazine.

[26]  Chih-Jen Lin,et al.  On the Convergence of Multiplicative Update Algorithms for Nonnegative Matrix Factorization , 2007, IEEE Transactions on Neural Networks.

[27]  Nicolas Gillis,et al.  Dimensionality reduction, classification, and spectral mixture analysis using nonnegative underapproximation , 2010, Defense + Commercial Sensing.

[28]  J. Huriot,et al.  Economics of Cities , 2000 .

[29]  Lixing Han,et al.  On the rate of convergence of the image space reconstruction algorithm , 2009 .

[30]  Paul Belleflamme,et al.  Industrial Organization: Markets and Strategies , 2010 .

[31]  Charles L. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[32]  Nicolas Gillis,et al.  Nonnegative Factorization and The Maximum Edge Biclique Problem , 2008, 0810.4225.

[33]  Inderjit S. Dhillon,et al.  Fast Newton-type Methods for the Least Squares Nonnegative Matrix Approximation Problem , 2007, SDM.

[34]  Hyunsoo Kim,et al.  Nonnegative Matrix Factorization Based on Alternating Nonnegativity Constrained Least Squares and Active Set Method , 2008, SIAM J. Matrix Anal. Appl..

[35]  Geir B. Asheim,et al.  Justifying Social Discounting: The Rank-Discounted Utilitarian Approach , 2010, J. Econ. Theory.

[36]  Demetri Terzopoulos,et al.  Image Analysis Using Multigrid Relaxation Methods , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Miguel A. Carreira-Perpinan,et al.  Dimensionality Reduction , 2011 .

[38]  Jacques-François Thisse,et al.  Economic Geography: The Integration of Regions and Nations , 2008 .

[39]  Ho Ngoc-Diep.  Nonnegative matrix factorizationalgorithms and applications , 2008 .

[40]  A. Brandt.  Guide to multigrid development , 1982 .

[41]  Yu-Jin Zhang,et al.  FastNMF: highly efficient monotonic fixed-point nonnegative matrix factorization algorithm with good applicability , 2009, J. Electronic Imaging.

[42]  Andrzej Cichocki,et al.  Fast Local Algorithms for Large Scale Nonnegative Matrix and Tensor Factorizations , 2009, IEICE Trans. Fundam. Electron. Commun. Comput. Sci..

[43]  V. P. Pauca,et al.  Nonnegative matrix factorization for spectral data analysis , 2006 .

[44]  Hamilton-Jacobi Equations,et al.  Multigrid Methods for , 2011 .

[45]  W. Pohlmeier,et al.  High frequency financial econometrics : recent developments , 2007 .

[46]  Paul Van Dooren,et al.  Descent methods for Nonnegative Matrix Factorization , 2008, ArXiv.

[47]  Nicolas Gillis,et al.  Dimensionality reduction, classification, and spectral mixture analysis using non-negative underapproximation , 2011 .

[48]  Donghui Chen,et al.  Nonnegativity constraints in numerical analysis , 2009, The Birth of Numerical Analysis.

[49]  J. Gabszewicz.  La différenciation des produits , 2006 .

[50]  Hiroshi Uno,et al.  Nested potentials and robust equilibria , 2011 .

[51]  Koen Decancq,et al.  Copula-Based Orderings of Multivariate Dependence , 2010 .

[52]  Andrzej Cichocki,et al.  Hierarchical ALS Algorithms for Nonnegative Matrix and 3D Tensor Factorization , 2007, ICA.

[53]  C. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[54]  Stefan M. Wild,et al.  Improving non-negative matrix factorizations through structured initialization , 2004, Pattern Recognit..

[55]  M. Daube-Witherspoon,et al.  An Iterative Image Space Reconstruction Algorthm Suitable for Volume ECT , 1986, IEEE Transactions on Medical Imaging.

[56]  Stephen A. Vavasis,et al.  On the Complexity of Nonnegative Matrix Factorization , 2007, SIAM J. Optim..

[57]  Yassine Lefouili,et al.  Leniency programs for multimarket firms: The effect of Amnesty Plus on cartel formation ☆ , 2012 .

[58]  Pierre Pestieau,et al.  The impact of a minimum pension on old age poverty and its budgetary cost. Evidence from Latin America , 2011 .

[59]  Daniel Bienstock,et al.  Potential Function Methods for Approximately Solving Linear Programming Problems: Theory and Practice , 2002 .

[60]  R. Amir.  Supermodularity and Complementarity in Economics: An Elementary Survey , 2003 .

[61]  Andrzej Cichocki,et al.  Nonnegative Matrix and Tensor Factorization T , 2007 .

[62]  Lars Stentoft,et al.  Multivariate Option Pricing with Time Varying Volatility and Correlations , 2010 .

[63]  Yousef Saad,et al.  Graph-Based Multilevel Dimensionality Reduction with Applications to Eigenfaces and Latent Semantic Indexing , 2008, 2008 Seventh International Conference on Machine Learning and Applications.

[64]  M. Jünger,et al.  50 Years of Integer Programming 1958-2008 - From the Early Years to the State-of-the-Art , 2010 .

[65]  Christian Gourieroux,et al.  Simulation-based econometric methods , 1996 .

[66]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[67]  William L. Briggs,et al.  A multigrid tutorial , 1987 .

[68]  Chih-Jen Lin,et al.  Projected Gradient Methods for Nonnegative Matrix Factorization , 2007, Neural Computation.