Higher-Order Markov Tag-Topic Models for Tagged Documents and Images

This paper studies the topic modeling problem of tagged documents and images. Higher-order relations among tagged documents and images are major and ubiquitous characteristics, and play positive roles in extracting reliable and interpretable topics. In this paper, we propose the tag-topic models (TTM) to depict such higher-order topic structural dependencies within the Markov random field (MRF) framework. First, we use the novel factor graph representation of latent Dirichlet allocation (LDA)-based topic models from the MRF perspective, and present an efficient loopy belief propagation (BP) algorithm for approximate inference and parameter estimation. Second, we propose the factor hypergraph representation of TTM, and focus on both pairwise and higher-order relation modeling among tagged documents and images. Efficient loopy BP algorithm is developed to learn TTM, which encourages the topic labeling smoothness among tagged documents and images. Extensive experimental results confirm the incorporation of higher-order relations to be effective in enhancing the overall topic modeling performance, when compared with current state-of-the-art topic models, in many text and image mining tasks of broad interests such as word and link prediction, document classification, and tag recommendation.

[1]  Bo Thiesson,et al.  Markov Topic Models , 2009, AISTATS.

[2]  David A. Forsyth,et al.  Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary , 2002, ECCV.

[3]  Richard Szeliski,et al.  A Comparative Study of Energy Minimization Methods for Markov Random Fields with Smoothness-Based Priors , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Andrew McCallum,et al.  Automating the Construction of Internet Portals with Machine Learning , 2000, Information Retrieval.

[5]  Jiming Liu,et al.  Coauthor Network Topic Models with Application to Expert Finding , 2010, 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology.

[6]  Chong Wang,et al.  Reading Tea Leaves: How Humans Interpret Topic Models , 2009, NIPS.

[7]  Steffen Bickel,et al.  Unsupervised prediction of citation influences , 2007, ICML '07.

[8]  Andrzej Bargiela,et al.  Probabilistic Topic Models for Learning Terminological Ontologies , 2010, IEEE Transactions on Knowledge and Data Engineering.

[9]  Yan Liu,et al.  Topic-link LDA: joint models of topic and author community , 2009, ICML '09.

[10]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[11]  David M. Blei,et al.  Hierarchical relational models for document networks , 2009, 0909.4331.

[12]  Deng Cai,et al.  Topic modeling with network regularization , 2008, WWW.

[13]  X. Jin.  Factor graphs and the Sum-Product Algorithm , 2002 .

[14]  Gustavo Carneiro,et al.  Supervised Learning of Semantic Classes for Image Annotation and Retrieval , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Ramesh Nallapati,et al.  Labeled LDA: A supervised topic model for credit attribution in multi-labeled corpora , 2009, EMNLP.

[16]  Michal Rosen-Zvi,et al.  Latent Topic Models for Hypertext , 2008, UAI.

[17]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[18]  Jiming Liu,et al.  Multirelational Topic Models , 2009, 2009 Ninth IEEE International Conference on Data Mining.

[19]  J. Lafferty,et al.  Mixed-membership models of scientific publications , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[20]  Thomas L. Griffiths,et al.  The Author-Topic Model for Authors and Documents , 2004, UAI.

[21]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[22]  Mark Steyvers,et al.  Finding scientific topics , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[23]  Jia Zeng,et al.  Markov Random Field-Based Statistical Character Structure Modeling for Handwritten Chinese Character Recognition , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Gregor Heinrich.  Parameter estimation for text analysis , 2009 .

[25]  Andrew McCallum,et al.  Expertise modeling for matching papers with reviewers , 2007, KDD '07.

[26]  Michael J. Black,et al.  Efficient Belief Propagation with Learned Higher-Order Markov Random Fields , 2006, ECCV.

[27]  Jia Zeng,et al.  Type-2 fuzzy hidden Markov models and their application to speech recognition , 2006, IEEE Transactions on Fuzzy Systems.

[28]  Ramesh Nallapati,et al.  Joint latent topic models for text and citations , 2008, KDD.

[29]  Hal Daumé,et al.  Markov Random Topic Fields , 2009, ACL/IJCNLP.

[30]  Jiming Liu,et al.  Learning Topic Models by Belief Propagation , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Pushmeet Kohli,et al.  Minimizing sparse higher order energy functions of discrete variables , 2009, CVPR.

[32]  Steffen Klamt,et al.  Hypergraphs and Cellular Networks , 2009, PLoS Comput. Biol..

[33]  Tomás Werner,et al.  A Linear Programming Approach to Max-Sum Problem: A Review , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Jia Zeng,et al.  Enhancing MEDLINE document clustering by incorporating MeSH semantic similarity , 2009, Bioinform..

[35]  Brendan J. Frey,et al.  A comparison of algorithms for inference and learning in probabilistic graphical models , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  G. Qiu.  Indexing chromatic and achromatic patterns for content-based colour image retrieval , 2002, Pattern Recognit..

[37]  Stan Z. Li,et al.  Markov Random Field Modeling in Image Analysis , 2001, Computer Science Workbench.

[38]  Jia Zeng,et al.  Type-2 Fuzzy Markov Random Fields and Their Application to Handwritten Chinese Character Recognition , 2008, IEEE Transactions on Fuzzy Systems.

[39]  Jia Zeng,et al.  Type-2 fuzzy Gaussian mixture models , 2008, Pattern Recognit..