Shared Components Topic Models

With a few exceptions, extensions to latent Dirichlet allocation (LDA) have focused on the distribution over topics for each document. Much less attention has been given to the underlying structure of the topics themselves. As a result, most topic models generate topics independently from a single underlying distribution and require millions of parameters, in the form of multinomial distributions over the vocabulary. In this paper, we introduce the Shared Components Topic Model (SCTM), in which each topic is a normalized product of a smaller number of underlying component distributions. Our model learns these component distributions and the structure of how to combine subsets of them into topics. The SCTM can represent topics in a much more compact representation than LDA and achieves better perplexity with fewer parameters.

[1]  John D. Lafferty,et al.  Correlated Topic Models , 2005, NIPS.

[2]  Eric P. Xing,et al.  Sparse Additive Generative Models of Text , 2011, ICML.

[3]  David M. Blei,et al.  Supervised Topic Models , 2007, NIPS.

[4]  Mark Dredze,et al.  Shared Components Topic Models with Application to Selectional Preference , 2011 .

[5]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[6]  Thomas L. Griffiths,et al.  Hierarchical Topic Models and the Nested Chinese Restaurant Process , 2003, NIPS.

[7]  Chong Wang,et al.  The Discrete Infinite Logistic Normal Distribution for Mixed-Membership Modeling , 2011, AISTATS.

[8]  Michael I. Jordan,et al.  Hierarchical Dirichlet Processes , 2006 .

[9]  Wei Li,et al.  Pachinko allocation: DAG-structured mixture models of topic correlations , 2006, ICML.

[10]  G. C. Wei,et al.  A Monte Carlo Implementation of the EM Algorithm and the Poor Man's Data Augmentation Algorithms , 1990 .

[11]  Max Welling Donald,et al.  Products of Experts , 2007 .

[12]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[13]  Thomas L. Griffiths,et al.  Infinite latent feature models and the Indian buffet process , 2005, NIPS.

[14]  Katherine A. Heller,et al.  A Nonparametric Bayesian Approach to Modeling Overlapping Clusters , 2007, AISTATS.

[15]  Wei Li,et al.  Mixtures of hierarchical topics with Pachinko allocation , 2007, ICML '07.

[16]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[17]  Ruslan Salakhutdinov,et al.  Evaluation methods for topic models , 2009, ICML '09.

[18]  Wei Li,et al.  Nonparametric Bayes Pachinko Allocation , 2007, UAI.

[19]  Xiaojin Zhu,et al.  TagLDA: Bringing a document structure knowledge into topic models , 2006 .

[20]  Chong Wang,et al.  The IBP Compound Dirichlet Process and its Application to Focused Topic Modeling , 2010, ICML.

[21]  Gregor Heinrich.  Parameter estimation for text analysis , 2009 .