We propose an exact slice sampler for Hierarchical Dirichlet process (HDP) and its associated mixture models (Teh et al., 2006). Although there are existing MCMC algorithms for sampling from the HDP, a slice sampler has been missing from the literature. Slice sampling is well-known for its desirable properties including its fast mixing and its natural potential for parallelization. On the other hand, the hierarchical nature of HDPs poses challenges to adopting a full-fledged slice sampler that automatically truncates all the infinite measures involved without ad-hoc modifications. In this work, we adopt the powerful idea of Bayesian variable augmentation to address this challenge. By introducing new latent variables, we obtain a full factorization of the joint distribution that is suitable for slice sampling. Our algorithm has several appealing features such as (1) fast mixing; (2) remaining exact while allowing natural truncation of the underlying infinite-dimensional measures, as in (Kalli et al., 2011), resulting in updates of only a finite number of necessary atoms and weights in each iteration; and (3) being naturally suited to parallel implementations. The underlying principle for joint factorization of the full likelihood is simple and can be applied to many other settings, such as designing sampling algorithms for general dependent Dirichlet process (DDP) models.
[1]
John W. Fisher,et al.
Parallel Sampling of HDPs using Sub-Cluster Splits
,
2014,
NIPS.
[2]
Michael I. Jordan,et al.
Hierarchical Dirichlet Processes
,
2006
.
[3]
Max Welling,et al.
Asynchronous Distributed Learning of Topic Models
,
2008,
NIPS.
[4]
Michael I. Jordan,et al.
A Sticky HDP-HMM With Application to Speaker Diarization
,
2009,
0905.2592.
[5]
Yee Whye Teh,et al.
Collapsed Variational Dirichlet Process Mixture Models
,
2007,
IJCAI.
[6]
Yongdai Kim,et al.
An Online Gibbs Sampler Algorithm for Hierarchical Dirichlet Processes Prior
,
2016,
ECML/PKDD.
[7]
Lancelot F. James,et al.
Gibbs Sampling Methods for Stick-Breaking Priors
,
2001
.
[8]
E. Xing,et al.
A HIERARCHICAL DIRICHLET PROCESS MIXTURE MODEL FOR HAPLOTYPE RECONSTRUCTION FROM MULTI-POPULATION DATA
,
2008,
0812.4648.
[9]
Chong Wang,et al.
A Split-Merge MCMC Algorithm for the Hierarchical Dirichlet Process
,
2012,
ArXiv.
[10]
J. Sethuraman.
A CONSTRUCTIVE DEFINITION OF DIRICHLET PRIORS
,
1991
.
[11]
Max Welling,et al.
Distributed Algorithms for Topic Models
,
2009,
J. Mach. Learn. Res..
[12]
Stephen G. Walker,et al.
Slice sampling mixture models
,
2011,
Stat. Comput..