A Neural Framework for Generalized Topic Models

Topic models for text corpora comprise a popular family of methods that have inspired many extensions to encode properties such as sparsity, interactions with covariates, and the gradual evolution of topics. In this paper, we combine certain motivating ideas behind variations on topic models with modern techniques for variational inference to produce a flexible framework for topic modeling that allows for rapid exploration of different models. We first discuss how our framework relates to existing models, and then demonstrate that it achieves strong performance, with the introduction of sparsity controlling the trade off between perplexity and topic coherence. We have released our code and preprocessing scripts to support easy future comparisons and exploration.

[1]  Dustin Tran,et al.  Edward: A library for probabilistic modeling, inference, and criticism , 2016, ArXiv.

[2]  Francis R. Bach,et al.  Online Learning for Latent Dirichlet Allocation , 2010, NIPS.

[3]  Samy Bengio,et al.  Generating Sentences from a Continuous Space , 2015, CoNLL.

[4]  Dustin Tran,et al.  Automatic Differentiation Variational Inference , 2016, J. Mach. Learn. Res..

[5]  E-Step.  Structural Topic Models for Open Ended Survey Responses , 2022 .

[6]  David M. Blei,et al.  Deep Exponential Families , 2014, AISTATS.

[7]  Timothy Baldwin,et al.  Automatic Evaluation of Topic Coherence , 2010, NAACL.

[8]  Eric P. Xing,et al.  Sparse Additive Generative Models of Text , 2011, ICML.

[9]  Andrew Gelman,et al.  Automatic Variational Inference in Stan , 2015, NIPS.

[10]  David M. Blei,et al.  Supervised Topic Models , 2007, NIPS.

[11]  Viet-An Nguyen,et al.  Lexical and Hierarchical Topic Regression , 2013, NIPS.

[12]  Michael I. Jordan,et al.  Bayesian Nonnegative Matrix Factorization with Stochastic Variational Inference , 2014, Handbook of Mixed Membership Models and Their Applications.

[13]  John D. Lafferty,et al.  Correlated Topic Models , 2005, NIPS.

[14]  Diego Marcheggiani,et al.  Discrete-State Variational Autoencoders for Joint Discovery and Factorization of Relations , 2016, TACL.

[15]  Philip Resnik,et al.  Tea Party in the House: A Hierarchical Ideal Point Topic Model and Its Application to Republican Legislators in the 112th Congress , 2015, ACL.

[16]  John D. Lafferty,et al.  Dynamic topic models , 2006, ICML.

[17]  Geoffrey E. Hinton,et al.  Replicated Softmax: an Undirected Topic Model , 2009, NIPS.

[18]  Noah A. Smith,et al.  Friendships, Rivalries, and Trysts: Characterizing Relations between Ideas in Texts , 2017, ACL.

[19]  Charles A. Sutton,et al.  Autoencoding Variational Inference For Topic Models , 2017, ICLR.

[20]  David M. Blei,et al.  Build, Compute, Critique, Repeat: Data Analysis with Latent Variable Models , 2014 .

[21]  Eric P. Xing,et al.  Staying Informed: Supervised and Semi-Supervised Multi-View Topical Analysis of Ideological Perspective , 2010, EMNLP.

[22]  Noah A. Smith,et al.  A Sparse and Adaptive Prior for Time-Dependent Model Parameters , 2013, ArXiv.

[23]  Thomas L. Griffiths,et al.  The nested chinese restaurant process and bayesian nonparametric inference of topic hierarchies , 2007, JACM.

[24]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[25]  Phil Blunsom,et al.  Neural Variational Inference for Text Processing , 2015, ICML.

[26]  Thomas L. Griffiths,et al.  The Author-Topic Model for Authors and Documents , 2004, UAI.

[27]  Ruslan Salakhutdinov,et al.  Evaluation methods for topic models , 2009, ICML '09.

[28]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[29]  Måns Magnusson,et al.  Pulling Out the Stops: Rethinking Stopword Removal for Topic Models , 2017, EACL.

[30]  Ali Taylan Cemgil,et al.  Bayesian Inference for Nonnegative Matrix Factorisation Models , 2009, Comput. Intell. Neurosci..

[31]  Andre Wibisono,et al.  Streaming Variational Bayes , 2013, NIPS.

[32]  Karol Gregor,et al.  Neural Variational Inference and Learning in Belief Networks , 2014, ICML.

[33]  Sanjoy Dasgupta,et al.  A Generalization of Principal Components Analysis to the Exponential Family , 2001, NIPS.

[34]  Timothy Baldwin,et al.  Machine Reading Tea Leaves: Automatically Evaluating Topic Coherence and Topic Model Quality , 2014, EACL.

[35]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Daan Wierstra,et al.  Stochastic Backpropagation and Approximate Inference in Deep Generative Models , 2014, ICML.

[37]  David M. Blei,et al.  The Generalized Reparameterization Gradient , 2016, NIPS.