Partitioned Tensor Factorizations for Learning Mixed Membership Models

We present an efficient algorithm for learning mixed membership models when the number of variables p is much larger than the number of hidden components k. This algorithm reduces the computational complexity of state-of-the-art tensor methods, which require decomposing an O ( p ) tensor, to factorizingO (p/k) sub-tensors each of size O ( k ) . In addition, we address the issue of negative entries in the empirical method of moments based estimators. We provide sufficient conditions under which our approach has provable guarantees. Our approach obtains competitive empirical results on both simulated and real data.

[1]  Anima Anandkumar,et al.  Fast Detection of Overlapping Communities via Online Tensor Methods on GPUs , 2013, ArXiv.

[2]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[3]  Max Welling,et al.  Positive tensor factorization , 2001, Pattern Recognit. Lett..

[4]  Jiawei Han,et al.  Robust Tensor Decomposition with Gross Corruption , 2014, NIPS.

[5]  P. Schönemann,et al.  A generalized solution of the orthogonal procrustes problem , 1966 .

[6]  Prateek Jain,et al.  Tensor vs. Matrix Methods: Robust Tensor Decomposition under Block Sparse Perturbations , 2015, AISTATS.

[7]  Percy Liang,et al.  Tensor Factorization via Matrix Factorization , 2015, AISTATS.

[8]  Nino Antulov-Fantulin,et al.  Learning from Incomplete Ratings using Nonlinear Multi-layer Semi-Nonnegative Matrix Factorization , 2017, ArXiv.

[9]  M. Rudelson,et al.  The smallest singular value of a random rectangular matrix , 2008, 0802.3956.

[10]  P. Donnelly,et al.  Association mapping in structured populations. , 2000, American journal of human genetics.

[11]  Anima Anandkumar,et al.  A Spectral Algorithm for Latent Dirichlet Allocation , 2012, Algorithmica.

[12]  J. Kruskal.  Three-way arrays: rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics , 1977 .

[13]  이주연,et al.  Latent Dirichlet Allocation (LDA) 모델 기반의 인공지능(A.I.) 기술 관련 연구 활동 및 동향 분석 , 2018 .

[14]  Victoria Stodden,et al.  When Does Non-Negative Matrix Factorization Give a Correct Decomposition into Parts? , 2003, NIPS.

[15]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[16]  S. Leurgans,et al.  A Decomposition for Three-Way Arrays , 1993, SIAM J. Matrix Anal. Appl..

[17]  Sham M. Kakade,et al.  Learning mixtures of spherical gaussians: moment methods and spectral decompositions , 2012, ITCS '13.

[18]  Xi Chen,et al.  Spectral Methods Meet EM: A Provably Optimal Algorithm for Crowdsourcing , 2014, J. Mach. Learn. Res..

[19]  Pierre Comon,et al.  Handbook of Blind Source Separation: Independent Component Analysis and Applications , 2010 .

[20]  Alexander J. Smola,et al.  Fast and Guaranteed Tensor Decomposition via Sketching , 2015, NIPS.

[21]  M. Brand,et al.  A Parallel Quadratic Programming Algorithm for Model Predictive Control , 2011 .

[22]  P. Donnelly,et al.  Inference of population structure using multilocus genotype data. , 2000, Genetics.

[23]  Antoine Souloumiac,et al.  Joint diagonalization: Is non-orthogonal always preferable to orthogonal? , 2009, 2009 3rd IEEE International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP).

[24]  A. P. Dawid,et al.  Maximum Likelihood Estimation of Observer Error‐Rates Using the EM Algorithm , 1979 .

[25]  Tamir Hazan,et al.  Non-negative tensor factorization with applications to statistics and computer vision , 2005, ICML.

[26]  Sayan Mukherjee,et al.  Fast Moment Estimation for Generalized Latent Dirichlet Models , 2016, Journal of the American Statistical Association.

[27]  Sayan Mukherjee,et al.  Efficient Learning of Graded Membership Models , 2017, ArXiv.

[28]  David P. Woodruff,et al.  Sublinear Time Orthogonal Tensor Decomposition , 2016, NIPS.

[29]  L. Finesso,et al.  Nonnegative matrix factorization and I-divergence alternating minimization☆ , 2004, math/0412070.

[30]  Daniel D. Lee,et al.  Multiplicative Updates for Nonnegative Quadratic Programming in Support Vector Machines , 2002, NIPS.

[31]  Tamara G. Kolda,et al.  On Tensors, Sparsity, and Nonnegative Factorizations , 2011, SIAM J. Matrix Anal. Appl..

[32]  M. Woodbury,et al.  Mathematical typology: a grade of membership technique for obtaining disease definition. , 1978, Computers and biomedical research, an international journal.

[33]  Anima Anandkumar,et al.  Tensor decompositions for learning latent variable models , 2012, J. Mach. Learn. Res..

[34]  Matthew Brand,et al.  Parallel quadratic programming for image processing , 2011, 2011 18th IEEE International Conference on Image Processing.

[35]  Haesun Park,et al.  Algorithms for nonnegative matrix and tensor factorizations: a unified view based on block coordinate descent framework , 2014, J. Glob. Optim..

[36]  E. Erosheva.  Comparing Latent Structures of the Grade of Membership, Rasch, and Latent Class Models , 2005 .