Self-paced and soft-weighted nonnegative matrix factorization for data representation

Abstract Nonnegative matrix factorization (NMF) has received intensive attention due to producing a parts-based representation of the data. However, because of the non-convexity of NMF models, these methods easily obtain a bad local solution. To alleviate this deficiency, this paper presents a novel NMF method by gradually including data points into NMF from easy to complex, namely self-paced learning (SPL), which is shown to be beneficial in avoiding a bad local solution. Furthermore, instead of using the conventional hard weighting scheme, we adopt the soft weighting strategy of SPL to further improve the performance of our model. An iterative updating algorithm is proposed to solve the optimization problem of our method. The convergence of the updating rules is also theoretically guaranteed. Experiments on both toy data and real-world benchmark datasets demonstrate the effectiveness of the proposed method.

[1]  Jason Weston,et al.  Curriculum learning , 2009, ICML '09.

[2]  Daphne Koller,et al.  Self-Paced Learning for Latent Variable Models , 2010, NIPS.

[3]  Zenglin Xu,et al.  Infinite Tucker Decomposition: Nonparametric Bayesian Models for Multiway Data Analysis , 2011, ICML.

[4]  Zenglin Xu,et al.  Robust Softmax Regression for Multi-class Classification with Self-Paced Learning , 2017, IJCAI.

[5]  Xuelong Li,et al.  Self-weighted Multiview Clustering with Multiple Graphs , 2017, IJCAI.

[6]  Hava T. Siegelmann,et al.  Support Vector Clustering , 2002, J. Mach. Learn. Res..

[7]  Zenglin Xu,et al.  Nonnegative matrix factorization with adaptive neighbors , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[8]  Daphne Koller,et al.  Learning specific-class segmentation from diverse data , 2011, 2011 International Conference on Computer Vision.

[9]  Ivor W. Tsang,et al.  Spectral Embedded Clustering: A Framework for In-Sample and Out-of-Sample Spectral Clustering , 2011, IEEE Transactions on Neural Networks.

[10]  Zenglin Xu,et al.  Balanced self-paced learning with feature corruption , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[11]  Feiping Nie,et al.  Clustering and projected clustering with adaptive neighbors , 2014, KDD.

[12]  Zenglin Xu,et al.  Variational Random Function Model for Network Modeling , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[13]  Shi-Jinn Horng,et al.  Matrix-based dynamic updating rough fuzzy approximations for data mining , 2017, Knowl. Based Syst..

[14]  Zenglin Xu,et al.  Association Discovery and Diagnosis of Alzheimer's Disease with Bayesian Multiview Learning , 2016, J. Artif. Intell. Res..

[15]  Zhang Yi,et al.  Matrix approach to decision-theoretic rough sets for evolving data , 2016, Knowl. Based Syst..

[16]  Feiping Nie,et al.  Robust Manifold Nonnegative Matrix Factorization , 2014, ACM Trans. Knowl. Discov. Data.

[17]  Fei-Fei Li,et al.  Shifting Weights: Adapting Object Detectors from Image to Video , 2012, NIPS.

[18]  Chris H. Q. Ding,et al.  Convex and Semi-Nonnegative Matrix Factorizations , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Zenglin Xu,et al.  Bayesian Nonparametric Models for Multiway Data Analysis , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[21]  Chris H. Q. Ding,et al.  Robust nonnegative matrix factorization using L21-norm , 2011, CIKM '11.

[22]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[23]  Yueting Zhuang,et al.  Self-Paced Boost Learning for Classification , 2016, IJCAI.

[24]  Michael W. Berry,et al.  Text Mining Using Non-Negative Matrix Factorizations , 2004, SDM.

[25]  Zenglin Xu,et al.  Knowledge Base Completion by Variational Bayesian Neural Tensor Decomposition , 2018, Cognitive Computation.

[26]  Fillia Makedon,et al.  Fast Nonnegative Matrix Tri-Factorization for Large-Scale Data Co-Clustering , 2011, IJCAI.

[27]  V. P. Pauca,et al.  Nonnegative matrix factorization for spectral data analysis , 2006 .

[28]  Licheng Jiao,et al.  A fast tri-factorization method for low-rank matrix recovery and completion , 2013, Pattern Recognit..

[29]  Xiangxiang Zhu,et al.  Improved self-paced learning framework for nonnegative matrix factorization , 2017, Pattern Recognit. Lett..

[30]  Carlotta Domeniconi,et al.  Weighted-object ensemble clustering: methods and analysis , 2016, Knowledge and Information Systems.

[31]  Zhao Kang,et al.  Kernel-driven similarity learning , 2017, Neurocomputing.

[32]  Seungjin Choi,et al.  Orthogonal nonnegative matrix tri-factorization for co-clustering: Multiplicative updates on Stiefel manifolds , 2010, Inf. Process. Manag..

[33]  Chris H. Q. Ding,et al.  Orthogonal nonnegative matrix t-factorizations for clustering , 2006, KDD '06.

[34]  Feiping Nie,et al.  The Constrained Laplacian Rank Algorithm for Graph-Based Clustering , 2016, AAAI.

[35]  J. MacQueen.  Some methods for classification and analysis of multivariate observations , 1967 .

[36]  Joydeep Ghosh,et al.  Cluster Ensembles --- A Knowledge Reuse Framework for Combining Multiple Partitions , 2002, J. Mach. Learn. Res..

[37]  Dingcheng Li,et al.  Spectral co-clustering ensemble , 2015, Knowl. Based Syst..

[38]  Deyu Meng,et al.  Easy Samples First: Self-paced Reranking for Zero-Example Multimedia Search , 2014, ACM Multimedia.

[39]  Zenglin Xu,et al.  Robust multi-view data clustering with multi-view capped-norm K-means , 2018, Neurocomputing.

[40]  Zenglin Xu,et al.  Self-weighted multi-view clustering with soft capped norm , 2018, Knowl. Based Syst..

[41]  Carlotta Domeniconi,et al.  Weighted-Object Ensemble Clustering , 2013, 2013 IEEE 13th International Conference on Data Mining.

[42]  Zenglin Xu,et al.  Adaptive Regularization for Transductive Support Vector Machine , 2009, NIPS.

[43]  Vipin Kumar,et al.  WebACE: a Web agent for document categorization and exploration , 1998, AGENTS '98.

[44]  Qi Xie,et al.  Self-Paced Learning for Matrix Factorization , 2015, AAAI.

[45]  Feiping Nie,et al.  Robust Capped Norm Nonnegative Matrix Factorization: Capped Norm NMF , 2015, CIKM.

[46]  Hamido Fujita,et al.  A framework for integrating a decision tree learning algorithm and cluster analysis , 2013, 2013 IEEE 12th International Conference on Intelligent Software Methodologies, Tools and Techniques (SoMeT).

[47]  Feiping Nie,et al.  Efficient and Robust Feature Selection via Joint ℓ2, 1-Norms Minimization , 2010, NIPS.

[48]  Qingyao Wu,et al.  NMFE-SSCC: Non-negative matrix factorization ensemble for semi-supervised collective classification , 2015, Knowl. Based Syst..

[49]  Zenglin Xu,et al.  Adaptive local structure learning for document co-clustering , 2018, Knowl. Based Syst..

[50]  Shiguang Shan,et al.  Self-Paced Curriculum Learning , 2015, AAAI.

[51]  Michael I. Jordan,et al.  On Spectral Clustering: Analysis and an algorithm , 2001, NIPS.

[52]  Zenglin Xu,et al.  Robust graph regularized nonnegative matrix factorization for clustering , 2017, Data Mining and Knowledge Discovery.