Design and evaluation of a parallel algorithm for inferring topic hierarchies

We propose a novel parallel Algorithm for inferring topic hierarchies using HLDA.We use loosely-coupled parallel tasks that do not require frequent synchronization.The parallel Algorithm is well-suited to be run on distributed computing systems.The proposed Algorithm achieves a predictive accuracy on par with that of HLDA.The parallel Algorithm exhibits a near-linear speed-up and scales well. The rapid growth of information in the digital world especially on the web, calls for automated methods of organizing the digital information for convenient access and efficient information retrieval. Topic modeling is a branch of machine learning and probabilistic graphical modeling that helps in arranging the web pages according to their topical structure. The topic distribution over a set of documents (web pages) and the affinity of a document toward a specific topic can be revealed using topic modeling. Topic modeling algorithms are typically computationally expensive due to their iterative nature. Recent research efforts have attempted to parallelize specific topic models and are successful in their attempts. These parallel algorithms however have tightly-coupled parallel processes which require frequent synchronization and are also tightly coupled with the underlying topic model which is used for inferring the topic hierarchy. In this paper, we propose a parallel algorithm to infer topic hierarchies from a large scale document corpus. A key feature of the proposed algorithm is that it exploits coarse grained parallelism and the components running in parallel need not synchronize after every iteration, thus the algorithm lends itself to be implemented on a geographically dispersed set of processing elements interconnected through a network. The parallel algorithm realizes a speed up of 53.5 on a 32-node cluster of dual-core workstations and at the same time achieving approximately the same likelihood or predictive accuracy as that of the sequential algorithm, with respect to the performance of Information Retrieval tasks.

[1]  Ronald L. Rivest,et al.  Introduction to Algorithms , 1990 .

[2]  Jun S. Liu,et al.  The Collapsed Gibbs Sampler in Bayesian Computations with Applications to a Gene Regulation Problem , 1994 .

[3]  S. Mercy Shalinie,et al.  Parallel algorithms for merging topic trees and their application in meta search engines , 2015, CODS.

[4]  Edward Y. Chang,et al.  PLDA: Parallel Latent Dirichlet Allocation for Large-Scale Applications , 2009, AAIM.

[5]  Max Welling,et al.  Distributed Algorithms for Topic Models , 2009, J. Mach. Learn. Res..

[6]  Alexander J. Smola,et al.  Scalable inference in latent variable models , 2012, WSDM '12.

[7]  Alexander J. Smola,et al.  An architecture for parallel topic models , 2010, Proc. VLDB Endow..

[8]  Padhraic Smyth,et al.  Scalable Parallel Topic Models , 2006 .

[9]  R. A. Leibler,et al.  On Information and Sufficiency , 1951 .

[10]  Sunil Arya,et al.  Space-time tradeoffs for approximate nearest neighbor searching , 2009, JACM.

[11]  Max Welling,et al.  Asynchronous Distributed Learning of Topic Models , 2008, NIPS.

[12]  Leonidas J. Guibas,et al.  The Earth Mover's Distance as a Metric for Image Retrieval , 2000, International Journal of Computer Vision.

[13]  Michael Werman,et al.  The Quadratic-Chi Histogram Distance Family , 2010, ECCV.

[14]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[15]  Lu Yang,et al.  Towards big topic modeling , 2017, Inf. Sci..

[16]  Nando de Freitas,et al.  An Introduction to MCMC for Machine Learning , 2004, Machine Learning.

[17]  Thomas L. Griffiths,et al.  The nested chinese restaurant process and bayesian nonparametric inference of topic hierarchies , 2007, JACM.