Recovery Guarantee of Non-negative Matrix Factorization via Alternating Updates

Non-negative matrix factorization is a popular tool for decomposing data into feature and weight matrices under non-negativity constraints. It enjoys practical success but is poorly understood theoretically. This paper proposes an algorithm that alternates between decoding the weights and updating the features, and shows that assuming a generative model of the data, it provably recovers the ground-truth under fairly mild conditions. In particular, its only essential requirement on features is linear independence. Furthermore, the algorithm uses ReLU to exploit the non-negativity for decoding the weights, and thus can tolerate adversarial noise that can potentially be as large as the signal, and can tolerate unbiased noise much larger than the signal. The analysis relies on a carefully designed coupling between two potential functions, which we believe is of independent interest.

[1]  Sanjeev Arora,et al.  Learning Topic Models -- Going beyond SVD , 2012, 2012 IEEE 53rd Annual Symposium on Foundations of Computer Science.

[2]  David J. Field,et al.  Sparse coding with an overcomplete basis set: A strategy employed by V1? , 1997, Vision Research.

[3]  Eric P. Xing,et al.  Sparse Topical Coding , 2011, UAI.

[4]  Venkatesh Saligrama,et al.  Topic Discovery through Data Dependent and Random Projections , 2013, ICML.

[5]  A. Föhrenbach,et al.  SIMPLE++ , 2000, OR Spectr..

[6]  Dmitri B. Chklovskii,et al.  A Hebbian/Anti-Hebbian network derived from online non-negative matrix factorization can cluster and discover sparse features , 2014, 2014 48th Asilomar Conference on Signals, Systems and Computers.

[7]  Tao Hu,et al.  A Hebbian/Anti-Hebbian network for online sparse dictionary learning derived from symmetric matrix factorization , 2014, 2014 48th Asilomar Conference on Signals, Systems and Computers.

[8]  Andrej Risteski,et al.  On some provably correct cases of variational inference for topic models , 2015, NIPS.

[9]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[10]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[11]  Dmitri B. Chklovskii,et al.  Optimization theory of Hebbian/anti-Hebbian networks for PCA and whitening , 2015, 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[12]  Chiranjib Bhattacharyya,et al.  A provable SVD-based algorithm for learning topics in dominant admixture corpus , 2014, NIPS.

[13]  Alan M. Frieze,et al.  Learning linear transformations , 1996, Proceedings of 37th Conference on Foundations of Computer Science.

[14]  Chiranjib Bhattacharyya,et al.  Non-negative Matrix Factorization under Heavy Noise , 2016, ICML.

[15]  R. Rosenfeld.  Nature , 2009, Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery.

[16]  Dmitri B. Chklovskii,et al.  A Normative Theory of Adaptive Dimensionality Reduction in Neural Networks , 2015, NIPS.

[17]  Sanjeev Arora,et al.  Provable ICA with Unknown Gaussian Noise, and Implications for Gaussian Mixtures and Autoencoders , 2012, Algorithmica.

[18]  Sanjeev Arora,et al.  A Practical Algorithm for Topic Modeling with Provable Guarantees , 2012, ICML.

[19]  Sanjeev Arora,et al.  Simple, Efficient, and Neural Algorithms for Sparse Coding , 2015, COLT.

[20]  H. Sebastian Seung,et al.  Unsupervised Learning by Convex and Conic Coding , 1996, NIPS.

[21]  David M. Blei,et al.  Probabilistic topic models , 2012, Commun. ACM.

[22]  Venkatesh Saligrama,et al.  Efficient Distributed Topic Modeling with Provable Guarantees , 2014, AISTATS.

[23]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[24]  Anima Anandkumar,et al.  Two SVDs Suffice: Spectral decompositions for probabilistic topic modeling and latent Dirichlet allocation , 2012, NIPS 2012.

[25]  Sanjeev Arora,et al.  Computing a nonnegative matrix factorization -- provably , 2011, STOC '12.