Convergence Analysis of Graph Regularized Non-Negative Matrix Factorization

Graph regularized non-negative matrix factorization (NMF) algorithms can be applied to information retrieval, image processing, and pattern recognition. However, challenge that still remains is to prove the convergence of this class of learning algorithms since the geometrical structure of the data space is considered. This paper presents the convergence properties of the graph regularized NMF learning algorithms. In the analysis, we focus on the study of Euclidian distance based algorithms. The structures of the fixed points are presented. The non-divergence of the learning algorithms is analyzed by constructing invariant sets for update rules. Based on Lyapunov indirect method, the stability of the algorithms is discussed in detail. The analysis shows that this class of NMF algorithms can converge to their fixed points under some given conditions. In the simulations, theoretical results presented in the paper are confirmed. For different initializations and data sets, variations of cost functions and decomposition data in the learning are presented to show the convergence features of the discussed NMF update rules, and the convergence speed of the algorithms is also investigated.

[1]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[2]  Jiawei Han,et al.  Non-negative Matrix Factorization on Manifold , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[3]  Yann LeCun,et al.  Dimensionality Reduction by Learning an Invariant Mapping , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[4]  Chih-Jen Lin,et al.  On the Convergence of Multiplicative Update Algorithms for Nonnegative Matrix Factorization , 2007, IEEE Transactions on Neural Networks.

[5]  Zhigang Luo,et al.  NeNMF: An Optimal Gradient Method for Nonnegative Matrix Factorization , 2012, IEEE Transactions on Signal Processing.

[6]  Zhang Yi,et al.  A Family of Fuzzy Learning Algorithms for Robust Principal Component Analysis Neural Networks , 2010, IEEE Transactions on Fuzzy Systems.

[7]  Mikhail Belkin,et al.  Laplacian Eigenmaps and Spectral Techniques for Embedding and Clustering , 2001, NIPS.

[8]  Zhigang Luo,et al.  Non-Negative Patch Alignment Framework , 2011, IEEE Transactions on Neural Networks.

[9]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[10]  Yin Zhang,et al.  Accelerating the Lee-Seung Algorithm for Nonnegative Matrix Factorization , 2005 .

[11]  Zhang Yi,et al.  Global Convergence of GHA Learning Algorithm With Nonzero-Approaching Adaptive Learning Rates , 2007, IEEE Transactions on Neural Networks.

[12]  Michael W. Berry,et al.  Algorithms and applications for approximate nonnegative matrix factorization , 2007, Comput. Stat. Data Anal..

[13]  Mao Ye,et al.  Global Minima Analysis of Lee and Seung’s NMF Algorithms , 2012, Neural Processing Letters.

[14]  Patrik O. Hoyer,et al.  Non-negative sparse coding , 2002, Proceedings of the 12th IEEE Workshop on Neural Networks for Signal Processing.

[15]  L. Finesso,et al.  Nonnegative matrix factorization and I-divergence alternating minimization☆ , 2004, math/0412070.

[16]  R. Plemmons,et al.  Optimality, computation, and interpretation of nonnegative matrix factorizations , 2004 .

[17]  John Shawe-Taylor,et al.  MahNMF: Manhattan Non-negative Matrix Factorization , 2012, ArXiv.

[18]  Emmanuel Vincent,et al.  Stability Analysis of Multiplicative Update Algorithms and Application to Nonnegative Matrix Factorization , 2010, IEEE Transactions on Neural Networks.

[19]  Wei Liu,et al.  Robust and Scalable Graph-Based Semisupervised Learning , 2012, Proceedings of the IEEE.

[20]  Zhang Yi,et al.  Convergence Analysis of a Class of HyvÄrinen–Oja's ICA Learning Algorithms With Constant Learning Rates , 2009, IEEE Transactions on Signal Processing.

[21]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[22]  Xiaojun Wu,et al.  Graph Regularized Nonnegative Matrix Factorization for Data Representation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  David J. Kriegman,et al.  From Few to Many: Illumination Cone Models for Face Recognition under Variable Lighting and Pose , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[24]  Zhigang Luo,et al.  Online Nonnegative Matrix Factorization With Robust Stochastic Approximation , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[25]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[26]  Xian-Sheng Hua,et al.  Ensemble Manifold Regularization , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Wei Liu,et al.  Large Graph Construction for Scalable Semi-Supervised Learning , 2010, ICML.

[28]  Patrik O. Hoyer,et al.  Non-negative Matrix Factorization with Sparseness Constraints , 2004, J. Mach. Learn. Res..