Heterogeneous Supervised Topic Models

Abstract Researchers in the social sciences are often interested in the relationship between text and an outcome of interest, where the goal is to both uncover latent patterns in the text and predict outcomes for unseen texts. To this end, this paper develops the heterogeneous supervised topic model (HSTM), a probabilistic approach to text analysis and prediction. HSTMs posit a joint model of text and outcomes to find heterogeneous patterns that help with both text analysis and prediction. The main benefit of HSTMs is that they capture heterogeneity in the relationship between text and the outcome across latent topics. To fit HSTMs, we develop a variational inference algorithm based on the auto-encoding variational Bayes framework. We study the performance of HSTMs on eight datasets and find that they consistently outperform related methods, including fine-tuned black-box models. Finally, we apply HSTMs to analyze news articles labeled with pro- or anti-tone. We find evidence of differing language used to signal a pro- and anti-tone.

[1]  David M. Blei,et al.  Text-Based Ideal Points , 2020, ACL.

[2]  Yoav Goldberg,et al.  Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? , 2020, ACL.

[3]  David M. Blei,et al.  Topic Modeling in Embedding Spaces , 2019, Transactions of the Association for Computational Linguistics.

[4]  Noah A. Smith,et al.  Is Attention Interpretable? , 2019, ACL.

[5]  James Zou,et al.  Analyzing Polarization in Social Media: Method and Application to Tweets on 21 Mass Shootings , 2019, NAACL.

[6]  Byron C. Wallace,et al.  Attention is not Explanation , 2019, NAACL.

[7]  Cynthia Rudin,et al.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead , 2018, Nature Machine Intelligence.

[8]  Dumitru Erhan,et al.  The (Un)reliability of saliency methods , 2017, Explainable AI.

[9]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[10]  Mark Dredze,et al.  Deep Dirichlet Multinomial Regression , 2018, NAACL.

[11]  Bhavana Dalvi,et al.  A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications , 2018, NAACL.

[12]  Shi Feng,et al.  Pathologies of Neural Models Make Interpretations Difficult , 2018, EMNLP.

[13]  Guillaume Desjardins,et al.  Understanding disentangling in β-VAE , 2018, ArXiv.

[14]  Ying Huang,et al.  Efficient Correlated Topic Modeling with Topic Embedding , 2017, KDD.

[15]  Noah A. Smith,et al.  A Neural Framework for Generalized Topic Models , 2017, ArXiv.

[16]  Timothy Baldwin,et al.  Topically Driven Neural Language Model , 2017, ACL.

[17]  Charles A. Sutton,et al.  Autoencoding Variational Inference For Topic Models , 2017, ICLR.

[18]  Phil Blunsom,et al.  Neural Variational Inference for Text Processing , 2015, ICML.

[19]  Noah A. Smith,et al.  The Media Frames Corpus: Annotations of Frames Across Issues , 2015, ACL.

[20]  Philip Resnik,et al.  Tea Party in the House: A Hierarchical Ideal Point Topic Model and Its Application to Republican Legislators in the 112th Congress , 2015, ACL.

[21]  Rajarshi Das,et al.  Gaussian LDA for Topic Models with Word Embeddings , 2015, ACL.

[22]  Heng Ji,et al.  A Novel Neural Topic Model and Its Supervised Extension , 2015, AAAI.

[23]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[24]  David G. Rand,et al.  Structural Topic Models for Open‐Ended Survey Responses , 2014, American Journal of Political Science.

[25]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[26]  Viet-An Nguyen,et al.  Lexical and Hierarchical Topic Regression , 2013, NIPS.

[27]  Justin Grimmer,et al.  Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts , 2013, Political Analysis.

[28]  Hugo Larochelle,et al.  A Neural Autoregressive Topic Model , 2012, NIPS.

[29]  Eric P. Xing,et al.  Sparse Additive Generative Models of Text , 2011, ICML.

[30]  Matt Taddy,et al.  Multinomial Inverse Regression for Text Analysis , 2010, 1012.2098.

[31]  Chong Wang,et al.  Reading Tea Leaves: How Humans Interpret Topic Models , 2009, NIPS.

[32]  Ramesh Nallapati,et al.  Labeled LDA: A supervised topic model for credit attribution in multi-labeled corpora , 2009, EMNLP.

[33]  Michael I. Jordan,et al.  DiscLDA: Discriminative Learning for Dimensionality Reduction and Classification , 2008, NIPS.

[34]  David M. Blei,et al.  Supervised Topic Models , 2007, NIPS.

[35]  John D. Lafferty,et al.  Dynamic topic models , 2006, ICML.

[36]  Thomas L. Griffiths,et al.  The Author-Topic Model for Authors and Documents , 2004, UAI.

[37]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[38]  Eric R. Ziegel,et al.  Generalized Linear Models , 2002, Technometrics.

[39]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[40]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[41]  E-Step.  Structural Topic Models for Open Ended Survey Responses , 2022 .