Efficient Correlated Topic Modeling with Topic Embedding

Correlated topic modeling has been limited to small model and problem sizes due to their high computational cost and poor scaling. In this paper, we propose a new model which learns compact topic embeddings and captures topic correlations through the closeness between the topic vectors. Our method enables efficient inference in the low-dimensional embedding space, reducing previous cubic or quadratic time complexity to linear w.r.t the topic size. We further speedup variational inference with a fast sampler to exploit sparsity of topic occurrence. Extensive experiments show that our approach is capable of handling model and data scales which are several orders of magnitude larger than existing correlation results, without sacrificing modeling quality by providing competitive or superior performance in document classification and retrieval.

[1]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[2]  D. Blei,et al.  The Discrete Innite Logistic Normal Distribution , 2011, 1103.4789.

[3]  Xiaojin Zhu,et al.  A Topic Model for Word Sense Disambiguation , 2007, EMNLP.

[4]  David Blei,et al.  Correlated Random Measures , 2015 .

[5]  Wei Li,et al.  Pachinko allocation: DAG-structured mixture models of topic correlations , 2006, ICML.

[6]  Chunyan Miao,et al.  Generative Topic Embedding: a Continuous Representation of Documents , 2016, ACL.

[7]  David M. Blei,et al.  Sparse stochastic inference for latent Dirichlet allocation , 2012, ICML.

[8]  Eric P. Xing,et al.  Large-scale Distributed Dependent Nonparametric Trees , 2015, ICML.

[9]  Eric P. Xing,et al.  Seeking The Truly Correlated Topic Posterior - on tight approximate inference of logistic-normal admixture model , 2007, AISTATS.

[10]  Eric P. Xing,et al.  Controllable Text Generation , 2017, ArXiv.

[11]  Wenhao Yu,et al.  Supplementary material , 2015 .

[12]  Chong Wang,et al.  Stochastic variational inference , 2012, J. Mach. Learn. Res..

[13]  Hagai Attias,et al.  Independent factor topic models , 2009, ICML '09.

[14]  Alexander J. Smola,et al.  Reducing the sampling complexity of topic models , 2014, KDD.

[15]  Zhiting Hu,et al.  Joint Embedding of Hierarchical Categories and Entities for Concept Categorization and Dataless Classification , 2016, COLING.

[16]  Michalis K. Titsias,et al.  Variational Learning of Inducing Variables in Sparse Gaussian Processes , 2009, AISTATS.

[17]  Hady Wirawan Lauw,et al.  Semantic visualization for spherical representation , 2014, KDD.

[18]  Regina Barzilay,et al.  Low-Rank Tensors for Scoring Dependency Structures , 2014, ACL.

[19]  Qiaozhu Mei,et al.  PTE: Predictive Text Embedding through Large-scale Heterogeneous Text Networks , 2015, KDD.

[20]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[21]  Eric P. Xing,et al.  Nonparametric Variational Auto-Encoders for Hierarchical Representation Learning , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[22]  Thomas L. Griffiths,et al.  The nested chinese restaurant process and bayesian nonparametric inference of topic hierarchies , 2007, JACM.

[23]  Wenguang Chen,et al.  WarpLDA: a Simple and Efficient O(1) Algorithm for Latent Dirichlet Allocation , 2015, ArXiv.

[24]  Eric P. Xing,et al.  Grounding Topic Models with Knowledge Bases , 2016, IJCAI.

[25]  Tie-Yan Liu,et al.  LightLDA: Big Topic Models on Modest Computer Clusters , 2014, WWW.

[26]  Di Jiang,et al.  Latent Topic Embedding , 2016, COLING.

[27]  Rajarshi Das,et al.  Gaussian LDA for Topic Models with Word Embeddings , 2015, ACL.

[28]  Hady Wirawan Lauw,et al.  Manifold Learning for Jointly Modeling Topic and Visualization , 2014, AAAI.

[29]  Wenguang Chen,et al.  WarpLDA: a Cache Efficient O(1) Algorithm for Latent Dirichlet Allocation , 2015, Proc. VLDB Endow..

[30]  Amr Ahmed,et al.  On Tight Approximate Inference of the Logistic-Normal Topic Admixture Model , 2007 .

[31]  Geoffrey E. Hinton,et al.  Replicated Softmax: an Undirected Topic Model , 2009, NIPS.

[32]  Chuang Gan,et al.  Recurrent Topic-Transition GAN for Visual Paragraph Generation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[33]  Jure Leskovec,et al.  node2vec: Scalable Feature Learning for Networks , 2016, KDD.

[34]  Zoubin Ghahramani,et al.  Pitfalls in the use of Parallel Inference for the Dirichlet Process , 2014, ICML.

[35]  Eric P. Xing,et al.  Entity Hierarchy Embedding , 2015, ACL.

[36]  Andrew Gordon Wilson,et al.  Stochastic Variational Deep Kernel Learning , 2016, NIPS.

[37]  John D. Lafferty,et al.  A correlated topic model of Science , 2007, 0708.3601.

[38]  Bo Zhang,et al.  Scalable Inference for Logistic-Normal Topic Models , 2013, NIPS.

[39]  Miguel Lázaro-Gredilla,et al.  Doubly Stochastic Variational Bayes for non-Conjugate Inference , 2014, ICML.

[40]  Eric P. Xing,et al.  On Unifying Deep Generative Models , 2017, ICLR.

[41]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[42]  Nematollah Batmanghelich,et al.  Nonparametric Spherical Topic Modeling with Word Embeddings , 2016, ACL.

[43]  Eric P. Xing,et al.  Dependent nonparametric trees for dynamic hierarchical clustering , 2014, NIPS.