A permutation-augmented sampler for DP mixture models

We introduce a new inference algorithm for Dirichlet process mixture models. While Gibbs sampling and variational methods focus on local moves, the new algorithm makes more global moves. This is done by introducing a permutation of the data points as an auxiliary variable. The algorithm is a blocked sampler which alternates between sampling the clustering and sampling the permutation. The key to the efficiency of this approach is that it is possible to use dynamic programming to consider all exponentially many clusterings consistent with a given permutation. We also show that random projections can be used to effectively sample the permutation. The result is a stochastic hill-climbing algorithm that yields burn-in times significantly smaller than those of collapsed Gibbs sampling.

[1]  T. Ferguson.  A Bayesian Analysis of Some Nonparametric Problems , 1973 .

[2]  C. Antoniak.  Mixtures of Dirichlet Processes with Applications to Bayesian Nonparametric Problems , 1974 .

[3]  Wang,et al.  Nonuniversal critical dynamics in Monte Carlo simulations. , 1987, Physical review letters.

[4]  W. Wong,et al.  The calculation of posterior distributions by data augmentation , 1987 .

[5]  M. West,et al.  Hyperparameter estimation in Dirichlet process mixture models , 1992 .

[6]  M. Escobar,et al.  Bayesian Density Estimation and Inference Using Mixtures , 1995 .

[7]  Miklós Simonovits,et al.  Random walks and an O*(n5) volume algorithm for convex bodies , 1997, Random Struct. Algorithms.

[8]  M. Simonovits,et al.  Random walks and an O * ( n 5 ) volume algorithm for convex bodies , 1997 .

[9]  J. Pitman,et al.  The two-parameter Poisson-Dirichlet distribution derived from a stable subordinator , 1997 .

[10]  Jun S. Liu,et al.  Parameter Expansion for Data Augmentation , 1999 .

[11]  Nir Friedman,et al.  Being Bayesian about Network Structure , 2000, UAI.

[12]  Lancelot F. James,et al.  Gibbs Sampling Methods for Stick-Breaking Priors , 2001 .

[13]  D. B. Dahl.  An improved merge-split sampler for conjugate dirichlet process mixture models , 2003 .

[14]  Roded Sharan,et al.  Bayesian haplo-type inference via the dirichlet process , 2004, ICML.

[15]  Radford M. Neal,et al.  A Split-Merge Markov chain Monte Carlo Procedure for the Dirichlet Process Mixture Model , 2004 .

[16]  Nir Friedman,et al.  Being Bayesian About Network Structure. A Bayesian Approach to Structure Discovery in Bayesian Networks , 2004, Machine Learning.

[17]  Antonio Torralba,et al.  Describing Visual Scenes using Transformed Dirichlet Processes , 2005, NIPS.

[18]  Katherine A. Heller,et al.  Bayesian hierarchical clustering , 2005, ICML.

[19]  J. Pitman.  Combinatorial Stochastic Processes , 2006 .

[20]  Michael I. Jordan,et al.  Hierarchical Dirichlet Processes , 2006 .

[21]  Michael I. Jordan,et al.  Variational inference for Dirichlet process mixtures , 2006 .

[22]  Hal Daumé,et al.  Fast search for Dirichlet process mixture models , 2007, AISTATS.

[23]  Roded Sharan,et al.  Bayesian haplo-type inference via the dirichlet process , 2004, ICML.

[24]  Yee Whye Teh,et al.  Collapsed Variational Dirichlet Process Mixture Models , 2007, IJCAI.