Topic Memory Networks for Short Text Classification

Many classification models work poorly on short texts due to data sparsity. To address this issue, we propose topic memory networks for short text classification with a novel topic memory mechanism to encode latent topic representations indicative of class labels. Different from most prior work that focuses on extending features with external knowledge or pre-trained topics, our model jointly explores topic inference and text classification with memory networks in an end-to-end manner. Experimental results on four benchmark datasets show that our model outperforms state-of-the-art models on short text classification, meanwhile generates coherent topics.

[1]  Xiang Zhang,et al.  Character-level Convolutional Networks for Text Classification , 2015, NIPS.

[2]  Gerlof Bouma,et al.  Normalized (pointwise) mutual information in collocation extraction , 2009 .

[3]  Patrick Paroubek,et al.  Twitter Based System: Using Twitter for Disambiguating Sentiment Ambiguous Adjectives , 2010, *SEMEVAL.

[4]  David M. Blei,et al.  Variational Inference: A Review for Statisticians , 2016, ArXiv.

[5]  Jin Wang,et al.  Combining Knowledge with Deep Convolutional Neural Networks for Short Text Classification , 2017, IJCAI.

[6]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[7]  Xiaohui Yan,et al.  A biterm topic model for short texts , 2013, WWW.

[8]  Alex Graves,et al.  Neural Turing Machines , 2014, ArXiv.

[9]  Kyunghyun Cho,et al.  Efficient Character-level Document Classification by Combining Convolution and Recurrent Layers , 2016, ArXiv.

[10]  Heng Ji,et al.  A Novel Neural Topic Model and Its Supervised Extension , 2015, AAAI.

[11]  Charles A. Sutton,et al.  Autoencoding Variational Inference For Topic Models , 2017, ICLR.

[12]  Paolo Ferragina,et al.  Classification of Short Texts by Deploying Topical Annotations , 2012, ECIR.

[13]  Zengchang Qin,et al.  Topic modeling of Chinese language beyond a bag-of-words , 2016, Comput. Speech Lang..

[14]  Kam-Fai Wong,et al.  Microblog Conversation Recommendation via Joint Modeling of Topics and Discourse , 2018, NAACL.

[15]  Yue Zhang,et al.  Improving Twitter Sentiment Classification Using Topic-Enriched Multi-Prototype Word Embeddings , 2016, AAAI.

[16]  Qiang Yang,et al.  Transferring topical knowledge from auxiliary long texts for short text clustering , 2011, CIKM '11.

[17]  Zi-Yi Dou,et al.  Capturing User and Product Information for Document Level Sentiment Analysis with Deep Memory Network , 2017, EMNLP.

[18]  Phil Blunsom,et al.  Neural Variational Inference for Text Processing , 2015, ICML.

[19]  Thomas Hofmann,et al.  Probabilistic Latent Semantic Indexing , 1999, SIGIR Forum.

[20]  Zenglin Xu,et al.  Neural Relational Topic Models for Scientific Article Analysis , 2018, CIKM.

[21]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[22]  Michael Röder,et al.  Exploring the Space of Topic Coherence Measures , 2015, WSDM.

[23]  Phil Blunsom,et al.  Discovering Discrete Latent Topics with Neural Variational Inference , 2017, ICML.

[24]  Christopher D. Manning,et al.  Baselines and Bigrams: Simple, Good Sentiment and Topic Classification , 2012, ACL.

[25]  Daan Wierstra,et al.  Stochastic Backpropagation and Approximate Inference in Deep Generative Models , 2014, ICML.

[26]  Robert V. Lindsey,et al.  A Phrase-Discovering Topic Model Using Hierarchical Pitman-Yor Processes , 2012, EMNLP.

[27]  Tomas Mikolov,et al.  Bag of Tricks for Efficient Text Classification , 2016, EACL.

[28]  Weinan Zhang,et al.  Advertising Keywords Recommendation for Short-Text Web Pages Using Wikipedia , 2012, TIST.

[29]  Wei Gao,et al.  Topic Extraction from Microblog Posts Using Conversation Structures , 2016, ACL.

[30]  Qingcai Chen,et al.  LCSTS: A Large Scale Chinese Short Text Summarization Dataset , 2015, EMNLP.

[31]  Lucy Vanderwende,et al.  Exploring Content Models for Multi-Document Summarization , 2009, NAACL.

[32]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[33]  Jason Weston,et al.  End-To-End Memory Networks , 2015, NIPS.

[34]  Elena Ferrari,et al.  EgoCentric: Ego Networks for Knowledge-based Short Text Classification , 2014, CIKM.

[35]  Jiajun Zhang,et al.  Exploiting Word Internal Structures for Generic Chinese Sentence Representation , 2017, EMNLP.

[36]  Dong Wang,et al.  Relation Classification via Recurrent Neural Network , 2015, ArXiv.

[37]  Yoon Kim,et al.  Convolutional Neural Networks for Sentence Classification , 2014, EMNLP.

[38]  Tiejun Zhao,et al.  Target-dependent Twitter Sentiment Classification , 2011, ACL.

[39]  Rajarshi Das,et al.  Gaussian LDA for Topic Models with Word Embeddings , 2015, ACL.

[40]  Cícero Nogueira dos Santos,et al.  Deep Convolutional Neural Networks for Sentiment Analysis of Short Texts , 2014, COLING.

[41]  Michael Granitzer,et al.  Explaining Topical Distances Using Word Embeddings , 2016, 2016 27th International Workshop on Database and Expert Systems Applications (DEXA).

[42]  Jing Li,et al.  Encoding Conversation Context for Neural Keyphrase Extraction from Microblog Posts , 2018, NAACL.

[43]  Rich Caruana,et al.  Overfitting in Neural Nets: Backpropagation, Conjugate Gradient, and Early Stopping , 2000, NIPS.

[44]  Erik Cambria,et al.  Targeted Aspect-Based Sentiment Analysis via Embedding Commonsense Knowledge into an Attentive LSTM , 2018, AAAI.

[45]  Jason Weston,et al.  Memory Networks , 2014, ICLR.

[46]  Franck Dernoncourt,et al.  Sequential Short-Text Classification with Recurrent and Convolutional Neural Networks , 2016, NAACL.

[47]  Xuanjing Huang,et al.  Adversarial Multi-task Learning for Text Classification , 2017, ACL.

[48]  Lidong Bing,et al.  Recurrent Attention Network on Memory for Aspect Sentiment Analysis , 2017, EMNLP.

[49]  Aixin Sun,et al.  Topic Modeling for Short Texts with Auxiliary Word Embeddings , 2016, SIGIR.

[50]  Ming Yang,et al.  Bidirectional Long Short-Term Memory Networks for Relation Classification , 2015, PACLIC.

[51]  Susumu Horiguchi,et al.  Learning to classify short and sparse text & web with hidden topics from large-scale data collections , 2008, WWW.

[52]  Yulan He,et al.  Extracting Topical Phrases from Clinical Documents , 2016, AAAI.

[53]  Chong Wang,et al.  TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency , 2016, ICLR.

[54]  Xuanjing Huang,et al.  FudanNLP: A Toolkit for Chinese Natural Language Processing , 2013, ACL.