Parallel Sampling of DP Mixture Models using Sub-Cluster Splits

We present an MCMC sampler for Dirichlet process mixture models that can be parallelized to achieve significant computational gains. We combine a non-ergodic, restricted Gibbs iteration with split/merge proposals in a manner that produces an ergodic Markov chain. Each cluster is augmented with two sub-clusters to construct likely split moves. Unlike some previous parallel samplers, the proposed sampler enforces the correct stationary distribution of the Markov chain without the need for finite approximations. Empirical results illustrate that the new sampler exhibits better convergence properties than current methods.

[1]  Michael I. Jordan,et al.  Variational inference for Dirichlet process mixtures , 2006 .

[2]  Radford M. Neal,et al.  A Split-Merge Markov chain Monte Carlo Procedure for the Dirichlet Process Mixture Model , 2004 .

[3]  Antonio Torralba,et al.  Describing Visual Scenes using Transformed Dirichlet Processes , 2005, NIPS.

[4]  J. Pitman.  Combinatorial Stochastic Processes , 2006 .

[5]  Erik B. Sudderth.  Graphical models for visual object recognition and tracking , 2006 .

[6]  J. Sethuraman.  A CONSTRUCTIVE DEFINITION OF DIRICHLET PRIORS , 1991 .

[7]  Radford M. Neal.  Bayesian Mixture Modeling , 1992 .

[8]  Ben Taskar,et al.  A permutation-augmented sampler for DP mixture models , 2007, ICML '07.

[9]  Radford M. Neal.  Markov Chain Sampling Methods for Dirichlet Process Mixture Models , 2000 .

[10]  S. MacEachern.  Estimating normal means with a conjugate style dirichlet process prior , 1994 .

[11]  Radford M. Neal,et al.  Splitting and merging components of a nonconjugate Dirichlet process mixture model , 2007 .

[12]  Thomas L. Griffiths,et al.  Hierarchical Topic Models and the Nested Chinese Restaurant Process , 2003, NIPS.

[13]  W. Eric L. Grimson,et al.  Construction of Dependent Dirichlet Processes based on Poisson Processes , 2010, NIPS.

[14]  Yee Whye Teh,et al.  Collapsed Variational Dirichlet Process Mixture Models , 2007, IJCAI.

[15]  Max Welling,et al.  Asynchronous Distributed Learning of Topic Models , 2008, NIPS.

[16]  Roded Sharan,et al.  Bayesian haplo-type inference via the dirichlet process , 2004, ICML.

[17]  D. B. Dahl.  An improved merge-split sampler for conjugate dirichlet process mixture models , 2003 .

[18]  P. Green,et al.  Modelling Heterogeneity With and Without the Dirichlet Process , 2001 .

[19]  Michael I. Jordan,et al.  Hierarchical Dirichlet Processes , 2006 .

[20]  Ryan P. Adams,et al.  ClusterCluster: Parallel Markov Chain Monte Carlo for Dirichlet Process Mixtures , 2013, ArXiv.

[21]  Roded Sharan,et al.  Bayesian haplo-type inference via the dirichlet process , 2004, ICML.

[22]  Y. Teh,et al.  MCMC for Normalized Random Measure Mixture Models , 2013, 1310.0595.

[23]  W. K. Hastings,et al.  Monte Carlo Sampling Methods Using Markov Chains and Their Applications , 1970 .

[24]  Matthew T. Harrison,et al.  A simple example of Dirichlet process mixture inconsistency for the number of components , 2013, NIPS.

[25]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[26]  M. Escobar,et al.  Bayesian Density Estimation and Inference Using Mixtures , 1995 .

[27]  H. Ishwaran,et al.  Exact and approximate sum representations for the Dirichlet process , 2002 .

[28]  G. Roberts,et al.  Retrospective Markov chain Monte Carlo methods for Dirichlet process hierarchical models , 2007, 0710.4228.

[29]  S. MacEachern,et al.  Estimating mixture of dirichlet process models , 1998 .

[30]  Eric P. Xing,et al.  Parallel Markov Chain Monte Carlo for Nonparametric Mixture Models , 2013, ICML.

[31]  S. MacEachern,et al.  A semiparametric Bayesian model for randomised block designs , 1996 .

[32]  Lancelot F. James,et al.  Gibbs Sampling Methods for Stick-Breaking Priors , 2001 .

[33]  T. Ferguson.  A Bayesian Analysis of Some Nonparametric Problems , 1973 .

[34]  Michael A. West,et al.  Hierarchical priors and mixture models, with applications in regression and density estimation , 2006 .