Mean field inference for the Dirichlet process mixture model

Abstract: We present a systematic study of several recently proposed methods of mean field inference for the Dirichlet process mixture (DPM) model. These methods provide approximations to the posterior distribution and are derived using the truncated stick-breaking representation and related approaches.We investigate their use in density estimation and cluster allocation and compare to Monte-Carlo results. Further, more specific topics include the general mathematical structure of the mean field approximation, the handling of the truncation level, the effect of including a prior on the concentration parameter α of the DPM model, the relationship between the proposed variants of the mean field approximation, and the connection to maximum a-posteriori estimation of the DPM model.

[1]  T. Ferguson.  A Bayesian Analysis of Some Nonparametric Problems , 1973 .

[2]  C. Antoniak.  Mixtures of Dirichlet Processes with Applications to Bayesian Nonparametric Problems , 1974 .

[3]  M. Mézard,et al.  Spin Glass Theory and Beyond , 1987 .

[4]  K. Roeder.  Density estimation with confidence sets exemplified by superclusters and voids in the galaxies , 1990 .

[5]  J. Sethuraman.  A CONSTRUCTIVE DEFINITION OF DIRICHLET PRIORS , 1991 .

[6]  M. Escobar.  Estimating Normal Means with a Dirichlet Process Prior , 1994 .

[7]  S. MacEachern.  Estimating normal means with a conjugate style dirichlet process prior , 1994 .

[8]  M. Escobar,et al.  Bayesian Density Estimation and Inference Using Mixtures , 1995 .

[9]  S. MacEachern,et al.  Estimating mixture of dirichlet process models , 1998 .

[10]  Michael I. Jordan,et al.  Improving the Mean Field Approximation Via the Use of Mixture Distributions , 1999, Learning in Graphical Models.

[11]  M. Escobar,et al.  Markov Chain Sampling Methods for Dirichlet Process Mixture Models , 2000 .

[12]  Lancelot F. James,et al.  Gibbs Sampling Methods for Stick-Breaking Priors , 2001 .

[13]  M. Opper,et al.  Advanced mean field methods: theory and practice , 2001 .

[14]  M. Opper,et al.  Comparing the Mean Field Method and Belief Propagation for Approximate Inference in MRFs , 2001 .

[15]  H. Ishwaran,et al.  Exact and approximate sum representations for the Dirichlet process , 2002 .

[16]  Michael,et al.  On a Class of Bayesian Nonparametric Estimates : I . Density Estimates , 2008 .

[17]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[18]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[19]  D. Titterington,et al.  Convergence properties of a general algorithm for calculating variational Bayesian estimates for a normal mixture model , 2006 .

[20]  Max Welling,et al.  Accelerated Variational Dirichlet Process Mixtures , 2006, NIPS.

[21]  Michael I. Jordan,et al.  Variational inference for Dirichlet process mixtures , 2006 .

[22]  Stephen G. Walker,et al.  Sampling the Dirichlet Mixture Model with Slices , 2006, Commun. Stat. Simul. Comput..

[23]  Yee Whye Teh,et al.  Collapsed Variational Inference for HDP , 2007, NIPS.

[24]  Yee Whye Teh,et al.  Collapsed Variational Dirichlet Process Mixture Models , 2007, IJCAI.

[25]  David M. Blei,et al.  Relative Performance Guarantees for Approximate Inference in Latent Dirichlet Allocation , 2008, NIPS.

[26]  Michael I. Jordan,et al.  Graphical Models, Exponential Families, and Variational Inference , 2008, Found. Trends Mach. Learn..