Inference in Supervised latent Dirichlet allocation

Supervised latent Dirichlet allocation (Supervised-LDA) [1] is a probabilistic topic model that can be used for classification. One of the advantages of Supervised-LDA over unsupervised LDA is that it can potentially learn topics that are inline with the class label. The variational Bayes algorithm proposed in [1] for inference in Supervised-LDA suffers from high computational complexity. To address this issue, we develop computationally efficient inference methods for Supervised-LDA. Specifically, we present collapsed variational Bayes and MAP inference for parameter estimation in Supervised-LDA. Additionally, we present computationally efficient inference methods to determine the label of unlabeled data. We provide an empirical evaluation of the classification performance and computational complexity (training as well as classification runtime) of different inference methods for the Supervised-LDA model and a classifier based on probabilistic latent semantic analysis.

[1]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[2]  T. Minka.  Estimating a Dirichlet distribution , 2012 .

[3]  Eric P. Xing,et al.  MedLDA: maximum margin supervised topic models for regression and classification , 2009, ICML '09.

[4]  Pietro Perona,et al.  A Bayesian hierarchical model for learning natural scene categories , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[5]  Matthew J. Beal.  Variational algorithms for approximate Bayesian inference , 2003 .

[6]  Andrew Zisserman,et al.  Scene Classification Via pLSA , 2006, ECCV.

[7]  Michael I. Jordan,et al.  DiscLDA: Discriminative Learning for Dimensionality Reduction and Classification , 2008, NIPS.

[8]  Shuang-Hong Yang,et al.  Dirichlet-Bernoulli Alignment: A Generative Model for Multi-Class Multi-Label Multi-Instance Corpora , 2009, NIPS.

[9]  Yee Whye Teh,et al.  A Collapsed Variational Bayesian Inference Algorithm for Latent Dirichlet Allocation , 2006, NIPS.

[10]  David B. Dunson,et al.  A Bayesian Model for Simultaneous Image Clustering, Annotation and Object Segmentation , 2009, NIPS.

[11]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[12]  Ramesh Nallapati,et al.  Labeled LDA: A supervised topic model for credit attribution in multi-labeled corpora , 2009, EMNLP.

[13]  Chong Wang,et al.  Simultaneous image classification and annotation , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Andrew McCallum,et al.  Topic Models Conditioned on Arbitrary Features with Dirichlet-multinomial Regression , 2008, UAI.

[15]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[16]  Éric Gaussier,et al.  Relation between PLSA and NMF and implications , 2005, SIGIR '05.

[17]  Andrew McCallum,et al.  Rethinking LDA: Why Priors Matter , 2009, NIPS.

[18]  Ata Kabán,et al.  On an equivalence between PLSI and LDA , 2003, SIGIR.

[19]  David M. Blei,et al.  Supervised Topic Models , 2007, NIPS.

[20]  Hang Li,et al.  Named entity mining from click-through data using weakly supervised latent dirichlet allocation , 2009, KDD.

[21]  Yee Whye Teh,et al.  On Smoothing and Inference for Topic Models , 2009, UAI.