Bayesian Repulsive Gaussian Mixture Model

ABSTRACT We develop a general class of Bayesian repulsive Gaussian mixture models that encourage well-separated clusters, aiming at reducing potentially redundant components produced by independent priors for locations (such as the Dirichlet process). The asymptotic results for the posterior distribution of the proposed models are derived, including posterior consistency and posterior contraction rate in the context of nonparametric density estimation. More importantly, we show that compared to the independent prior on the component centers, the repulsive prior introduces additional shrinkage effect on the tail probability of the posterior number of components, which serves as a measurement of the model complexity. In addition, a generalized urn model that allows a random number of components and correlated component centers is developed based on the exchangeable partition distribution, which gives rise to the corresponding blocked-collapsed Gibbs sampler for posterior inference. We evaluate the performance and demonstrate the advantages of the proposed methodology through extensive simulation studies and real data analysis. Supplementary materials for this article are available online.

[1]  L. Schwartz.  On Bayes procedures , 1965 .

[2]  D. Blackwell,et al.  Ferguson Distributions Via Polya Urn Schemes , 1973 .

[3]  C. Antoniak.  Mixtures of Dirichlet Processes with Applications to Bayesian Nonparametric Problems , 1974 .

[4]  B. Silverman.  Density estimation for statistics and data analysis , 1986 .

[5]  Bernard W. Silverman,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[6]  L. Pettit.  The Conditional Predictive Ordinate for the Normal Distribution , 1990 .

[7]  W. Wong,et al.  Probability inequalities for likelihood ratios and convergence rates of sieve MLEs , 1995 .

[8]  P. Green.  Reversible jump Markov chain Monte Carlo computation and Bayesian model determination , 1995 .

[9]  P. Green,et al.  On Bayesian Analysis of Mixtures with an Unknown Number of Components (with discussion) , 1997 .

[10]  S. MacEachern,et al.  Estimating mixture of dirichlet process models , 1998 .

[11]  J. Ghosh.  Bayesian density estimation. , 1998 .

[12]  J. Ghosh,et al.  POSTERIOR CONSISTENCY OF DIRICHLET MIXTURES IN DENSITY ESTIMATION , 1999 .

[13]  A. Gordaliza,et al.  Robustness Properties of k Means and Trimmed k Means , 1999 .

[14]  Radford M. Neal.  Markov Chain Sampling Methods for Dirichlet Process Mixture Models , 2000 .

[15]  A. V. D. Vaart,et al.  Convergence rates of posterior distributions , 2000 .

[16]  M. Escobar,et al.  Markov Chain Sampling Methods for Dirichlet Process Mixture Models , 2000 .

[17]  Lancelot F. James,et al.  Gibbs Sampling Methods for Stick-Breaking Priors , 2001 .

[18]  A. V. D. Vaart,et al.  Entropies and rates of convergence for maximum likelihood and Bayes estimation for mixtures of normal densities , 2001 .

[19]  Lancelot F. James,et al.  Approximate Dirichlet Process Computing in Finite Normal Mixtures , 2002 .

[20]  S. Walker,et al.  On rates of convergence for posterior distributions , 2004 .

[21]  P. Müller,et al.  Bayesian inference for gene expression and proteomics , 2006 .

[22]  D. B. Dahl.  Bayesian Inference for Gene Expression and Proteomics: Model-Based Clustering for Expression Data via a Dirichlet Process Mixture Model , 2006 .

[23]  P. Müller,et al.  10 Model-Based Clustering for Expression Data via a Dirichlet Process Mixture Model , 2006 .

[24]  Stephen G. Walker,et al.  Sampling the Dirichlet Mixture Model with Slices , 2006, Commun. Stat. Simul. Comput..

[25]  A. Fearnside.  Bayesian analysis of finite mixture distributions using the allocation sampler , 2007 .

[26]  Z. Q. John Lu.  Bayesian Inference for Gene Expression and Proteomics , 2007 .

[27]  S. Walker,et al.  On rates of convergence for posterior distributions in infinite-dimensional models , 2007, 0708.1892.

[28]  S. Ghosal,et al.  Kullback Leibler property of kernel mixture priors in Bayesian density estimation , 2007, 0710.2746.

[29]  Yuefeng Wu,et al.  The L1-consistency of Dirichlet mixtures in multivariate Bayesian density estimation , 2010, J. Multivar. Anal..

[30]  S. Muhammad.  Bayesian Analysis Of Mixture Distributions , 2010 .

[31]  Yuhong Yang,et al.  Maximum Lq-likelihood estimation. , 2010, 1002.4533.

[32]  A. V. D. Vaart,et al.  Adaptive Bayesian density estimation with location-scale mixtures , 2010 .

[33]  Catia Scricciolo,et al.  Posterior rates of convergence for Dirichlet mixtures of exponential power densities , 2011 .

[34]  David B. Dunson,et al.  Repulsive Mixtures , 2012, NIPS.

[35]  Carey E. Priebe,et al.  Journal of the American Statistical Association Maximum Lq-likelihood Estimation via the Expectation-maximization Algorithm: a Robust Estimation of Mixture Models Maximum Lq-likelihood Estimation via the Expectation-maximization Algorithm: a Robust Estimation of Mixture Models , 2022 .

[36]  Matthew T. Harrison,et al.  A simple example of Dirichlet process mixture inconsistency for the number of components , 2013, NIPS.

[37]  S. Ghosal,et al.  Adaptive Bayesian multivariate density estimation with Dirichlet mixtures , 2011, 1109.6406.

[38]  A. Canale,et al.  Posterior asymptotics of nonparametric location-scale mixtures for multivariate density estimation , 2013, 1306.2671.

[39]  Jiahua Chen.  Consistency of the MLE under mixture models , 2016, 1607.01251.

[40]  Peter Müller,et al.  Bayesian inference for latent biologic structure with determinantal point processes (DPP) , 2015, Biometrics.

[41]  F. Quintana,et al.  Parsimonious Hierarchical Modeling Using Repulsive Distributions , 2017, 1701.04457.

[42]  Jeffrey W. Miller,et al.  Mixture Models With a Prior on the Number of Components , 2015, Journal of the American Statistical Association.

[43]  M. Steel,et al.  On choosing mixture components via non‐local priors , 2016, Journal of the Royal Statistical Society: Series B (Statistical Methodology).