Convergence Analysis of Single Latent Factor-Dependent, Nonnegative, and Multiplicative Update-Based Nonnegative Latent Factor Models

A single latent factor (LF)-dependent, nonnegative, and multiplicative update (SLF-NMU) learning algorithm is highly efficient in building a nonnegative LF (NLF) model defined on a high-dimensional and sparse (HiDS) matrix. However, convergence characteristics of such NLF models are never justified in theory. To address this issue, this study conducts rigorous convergence analysis for an SLF-NMU-based NLF model. The main idea is twofold: 1) proving that its learning objective keeps nonincreasing with its SLF-NMU-based learning rules via constructing specific auxiliary functions; and 2) proving that it converges to a stable equilibrium point with its SLF-NMU-based learning rules via analyzing the Karush–Kuhn–Tucker (KKT) conditions of its learning objective. Experimental results on ten HiDS matrices from real applications provide numerical evidence that indicates the correctness of the achieved proof.

[1]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[2]  Yu-Jin Zhang,et al.  Nonnegative Matrix Factorization: A Comprehensive Review , 2013, IEEE Transactions on Knowledge and Data Engineering.

[3]  Zhi-Quan Luo,et al.  Guaranteed Matrix Completion via Non-Convex Factorization , 2014, IEEE Transactions on Information Theory.

[4]  Xiaochun Cao,et al.  A Unified Semi-Supervised Community Detection Framework Using Latent Space Graph Regularization , 2015, IEEE Transactions on Cybernetics.

[5]  Yair Carmon,et al.  Accelerated Methods for NonConvex Optimization , 2018, SIAM J. Optim..

[6]  Mark W. Schmidt,et al.  Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition , 2016, ECML/PKDD.

[7]  MengChu Zhou,et al.  An Efficient Second-Order Approach to Factorize Sparse Matrices in Recommender Systems , 2015, IEEE Transactions on Industrial Informatics.

[8]  Martin Ester,et al.  A matrix factorization technique with trust propagation for recommendation in social networks , 2010, RecSys '10.

[9]  Fei Wang,et al.  Community discovery using nonnegative matrix factorization , 2011, Data Mining and Knowledge Discovery.

[10]  MengChu Zhou,et al.  Centralized Charging Strategy and Scheduling Algorithm for Electric Vehicles Under a Battery Swapping Scenario , 2016, IEEE Transactions on Intelligent Transportation Systems.

[11]  Nikos D. Sidiropoulos,et al.  A Flexible and Efficient Algorithmic Framework for Constrained Matrix and Tensor Factorization , 2015, IEEE Transactions on Signal Processing.

[12]  Shuai Li,et al.  Symmetric and Nonnegative Latent Factor Models for Undirected, High-Dimensional, and Sparse Networks in Industrial Applications , 2017, IEEE Transactions on Industrial Informatics.

[13]  George Trigeorgis,et al.  A Deep Matrix Factorization Method for Learning Attribute Representations , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Chih-Jen Lin,et al.  Projected Gradient Methods for Nonnegative Matrix Factorization , 2007, Neural Computation.

[15]  MengChu Zhou,et al.  Latent Factor-Based Recommenders Relying on Extended Stochastic Gradient Descent Algorithms , 2019, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[16]  Neil J. Hurley,et al.  Novelty and Diversity in Top-N Recommendation -- Analysis and Evaluation , 2011, TOIT.

[17]  Yannick Deville,et al.  Linear-Quadratic Blind Source Separation Using NMF to Unmix Urban Hyperspectral Images , 2014, IEEE Transactions on Signal Processing.

[18]  Vincent Y. F. Tan,et al.  A Unified Convergence Analysis of the Multiplicative Update Algorithm for Regularized Nonnegative Matrix Factorization , 2016, IEEE Transactions on Signal Processing.

[19]  Andrew M. Gross,et al.  Network-based stratification of tumor mutations , 2013, Nature Methods.

[20]  Zoubin Ghahramani,et al.  Probabilistic machine learning and artificial intelligence , 2015, Nature.

[21]  Bradley N. Miller,et al.  GroupLens: applying collaborative filtering to Usenet news , 1997, CACM.

[22]  MengChu Zhou,et al.  An Efficient Non-Negative Matrix-Factorization-Based Approach to Collaborative Filtering for Recommender Systems , 2014, IEEE Transactions on Industrial Informatics.

[23]  Mao Ye,et al.  Convergence Analysis of Graph Regularized Non-Negative Matrix Factorization , 2014, IEEE Transactions on Knowledge and Data Engineering.

[24]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[25]  Zhaoshui He,et al.  Symmetric Nonnegative Matrix Factorization: Algorithms and Applications to Probabilistic Clustering , 2011, IEEE Transactions on Neural Networks.

[26]  Nathan Srebro,et al.  Global Optimality of Local Search for Low Rank Matrix Recovery , 2016, NIPS.

[27]  P. Paatero,et al.  Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values† , 1994 .

[28]  Zibin Zheng,et al.  QoS-Aware Web Service Recommendation by Collaborative Filtering , 2011, IEEE Transactions on Services Computing.

[29]  Wanyuan Wang,et al.  Community-Aware Task Allocation for Social Networked Multiagent Systems , 2014, IEEE Transactions on Cybernetics.

[30]  Hyunsoo Kim,et al.  Sparse Non-negative Matrix Factorizations via Alternating Non-negativity-constrained Least Squares , 2006 .

[31]  John D. Lafferty,et al.  Convergence Analysis for Rectangular Matrix Completion Using Burer-Monteiro Factorization and Gradient Descent , 2016, ArXiv.

[32]  Chih-Jen Lin,et al.  A fast parallel SGD for matrix factorization in shared memory systems , 2013, RecSys.

[33]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[34]  Michael R. Lyu,et al.  Learning to recommend with social trust ensemble , 2009, SIGIR.

[35]  Andrzej Cichocki,et al.  Multilayer Nonnegative Matrix Factorization Using Projected Gradient Approaches , 2007, Int. J. Neural Syst..

[36]  Keith C. C. Chan,et al.  Learning Latent Factors for Community Identification and Summarization , 2018, IEEE Access.

[37]  Jie Cao,et al.  Weighted modularity optimization for crisp and fuzzy community detection in large-scale networks , 2016 .

[38]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[39]  Davide Heller,et al.  STRING v10: protein–protein interaction networks, integrated over the tree of life , 2014, Nucleic Acids Res..

[40]  Hui Xiong,et al.  SAIL: Summation-bAsed Incremental Learning for Information-Theoretic Text Clustering , 2013, IEEE Transactions on Cybernetics.

[41]  Michael I. Jordan,et al.  Gradient Descent Only Converges to Minimizers , 2016, COLT.

[42]  Bir Bhanu,et al.  Semantic Concept Co-Occurrence Patterns for Image Annotation and Retrieval , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Zibin Zheng,et al.  Predicting Quality of Service for Selection by Neighborhood-Based Collaborative Filtering , 2013, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[44]  Norikazu Takahashi,et al.  Global convergence of modified multiplicative updates for nonnegative matrix factorization , 2013, Computational Optimization and Applications.

[45]  MengChu Zhou,et al.  An Incremental-and-Static-Combined Scheme for Matrix-Factorization-Based Collaborative Filtering , 2016, IEEE Transactions on Automation Science and Engineering.

[46]  Xiaojun Wu,et al.  Graph Regularized Nonnegative Matrix Factorization for Data Representation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Lin Wu,et al.  A Fast Algorithm for Nonnegative Matrix Factorization and Its Convergence , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[48]  Mohammad Ali Abbasi,et al.  Trust-Aware Recommender Systems , 2014 .

[49]  Hareton K. N. Leung,et al.  A Highly Efficient Approach to Protein Interactome Mapping Based on Collaborative Filtering Framework , 2015, Scientific Reports.

[50]  Maoguo Gong,et al.  A Multiobjective Sparse Feature Learning Model for Deep Neural Networks , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[51]  Tengyu Ma,et al.  Finding approximate local minima faster than gradient descent , 2016, STOC.

[52]  Justin K. Romberg,et al.  Convergence and Rate Analysis of Neural Networks for Sparse Approximation , 2011, IEEE Transactions on Neural Networks and Learning Systems.

[53]  Jong-Hwan Lee,et al.  A constrained alternating least squares nonnegative matrix factorization algorithm enhances task-related neuronal activity detection from single subject's fMRI data , 2011, 2011 International Conference on Machine Learning and Cybernetics.

[54]  Ke Ma,et al.  On Global Linear Convergence in Stochastic Nonconvex Optimization for Semidefinite Programming , 2019, IEEE Transactions on Signal Processing.

[55]  Michael I. Jordan,et al.  How to Escape Saddle Points Efficiently , 2017, ICML.

[56]  Junbin Gao,et al.  Dual Graph Regularized Latent Low-Rank Representation for Subspace Clustering , 2015, IEEE Transactions on Image Processing.

[57]  Mingsheng Shang,et al.  An Instance-Frequency-Weighted Regularization Scheme for Non-Negative Latent Factor Analysis on High-Dimensional and Sparse Data , 2021, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[58]  Jia Chen,et al.  Randomized latent factor model for high-dimensional and sparse matrices from industrial applications , 2018, 2018 IEEE 15th International Conference on Networking, Sensing and Control (ICNSC).

[59]  Yehuda Koren,et al.  Matrix Factorization Techniques for Recommender Systems , 2009, Computer.

[60]  Alexandros Nanopoulos,et al.  Modeling Users Preference Dynamics and Side Information in Recommender Systems , 2016, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[61]  Xin Liu,et al.  Document clustering based on non-negative matrix factorization , 2003, SIGIR.

[62]  Sheng Li,et al.  Deep Collaborative Filtering via Marginalized Denoising Auto-encoder , 2015, CIKM.

[63]  Robert M. Davison,et al.  Technical opinionWhy eBay lost to TaoBao in China: the Glocal advantage , 2009, CACM.

[64]  Christopher M. Kellett,et al.  Convergence Properties for Discrete-Time Nonlinear Systems , 2016, IEEE Transactions on Automatic Control.

[65]  MengChu Zhou,et al.  Generating Highly Accurate Predictions for Missing QoS Data via Aggregating Nonnegative Latent Factor Models , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[66]  Mingsheng Shang,et al.  A Fast Non-Negative Latent Factor Model Based on Generalized Momentum Method , 2021, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[67]  Yinghuan Shi,et al.  Incomplete-Data Oriented Multiview Dimension Reduction via Sparse Low-Rank Representation , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[68]  MengChu Zhou,et al.  Improved Quantum-Inspired Evolutionary Algorithm for Large-Size Lane Reservation , 2015, IEEE Transactions on Systems, Man, and Cybernetics: Systems.