Markov Chain Monte Carlo Methods and the Label Switching Problem in Bayesian Mixture Modeling

In the past ten years there has been a dramatic increase of in terest in the Bayesian analysis of finite mixture models. This is primarily because of the emergence of Markov chain Monte Carlo (MCMC) methods. While MCMC provides a convenient way to draw inference from compli cated statistical models, there are many, perhaps underappreciated, problems associated with the MCMC analysis of mixtures. The problems are mainly caused by the nonidentifiability of the components under symmetric priors, which leads to so-called label switching in the MCMC output. This means that ergodic averages of component specific quantities will be identical and thus useless for inference. We review the solutions to the label switching problem, such as artificial identifiability constraints, relabelling algorithms and label invariant loss functions. We also review various MCMC sampling schemes that have been suggested for mixture models and discuss posterior sensitivity to prior specification.

[1]  S. Newcomb.  A Generalized Theory of the Combination of Observations so as to Obtain the Best Result , 1886 .

[2]  K. Pearson.  Contributions to the Mathematical Theory of Evolution , 1894 .

[3]  C. R. Rao,et al.  The Utilization of Multiple Measurements in Problems of Biological Classification , 1948 .

[4]  D. Lindley.  A STATISTICAL PARADOX , 1957 .

[5]  M. Bartlett.  A comment on D. V. Lindley's statistical paradox , 1957 .

[6]  L. Baum,et al.  Statistical Inference for Probabilistic Functions of Finite State Markov Chains , 1966 .

[7]  W. K. Hastings,et al.  Monte Carlo Sampling Methods Using Markov Chains and Their Applications , 1970 .

[8]  L. Baum,et al.  A Maximization Technique Occurring in the Statistical Analysis of Probabilistic Functions of Markov Chains , 1970 .

[9]  M. Postman,et al.  Probes of large-scale structure in the Corona Borealis region. , 1986 .

[10]  A. F. Smith,et al.  Statistical analysis of finite mixture distributions , 1986 .

[11]  K. Roeder.  Density estimation with confidence sets exemplified by superclusters and voids in the galaxies , 1990 .

[12]  C. Robert,et al.  Estimation of Finite Mixture Distributions Through Bayesian Sampling , 1994 .

[13]  L. Tierney.  Markov Chains for Exploring Posterior Distributions , 1994 .

[14]  P. Green.  Reversible jump Markov chain Monte Carlo computation and Bayesian model determination , 1995 .

[15]  M. Escobar,et al.  Bayesian Density Estimation and Inference Using Mixtures , 1995 .

[16]  Radford M. Neal.  Sampling from multimodal distributions using tempered transitions , 1996, Stat. Comput..

[17]  P. Green,et al.  Corrigendum: On Bayesian analysis of mixtures with an unknown number of components , 1997 .

[18]  P. Green,et al.  On Bayesian Analysis of Mixtures with an Unknown Number of Components (with discussion) , 1997 .

[19]  P. Saama.  MAXIMUM LIKELIHOOD AND BAYESIAN METHODS FOR MIXTURES OF NORMAL DISTRIBUTIONS , 1997 .

[20]  Gilles Celeux,et al.  Bayesian Inference for Mixture: The Label Switching Problem , 1998, COMPSTAT.

[21]  C. Robert,et al.  MCMC Control Spreadsheets for Exponential Mixture Estimation , 1999 .

[22]  C. Robert,et al.  Bayesian inference in hidden Markov models through the reversible jump Markov chain Monte Carlo method , 2000 .

[23]  P. Green,et al.  Trans-dimensional Markov chain Monte Carlo , 2000 .

[24]  Hoon Kim,et al.  Monte Carlo Statistical Methods , 2000, Technometrics.

[25]  M. Stephens.  Bayesian analysis of mixture models with an unknown number of components- an alternative to reversible jump methods , 2000 .

[26]  M. Stephens.  Dealing with label switching in mixture models , 2000 .

[27]  C. Robert,et al.  Computational and Inferential Difficulties with Mixture Posterior Distributions , 2000 .

[28]  Geoffrey J. McLachlan,et al.  Finite Mixture Models , 2019, Annual Review of Statistics and Its Application.

[29]  Christian P. Robert,et al.  Reversible Jump MCMC Converging to Birth-and-Death MCMC and More General Continuous Time Samplers , 2001 .

[30]  Carl E. Rasmussen,et al.  Factorial Hidden Markov Models , 1997 .

[31]  S. Frühwirth-Schnatter.  Markov chain Monte Carlo Estimation of Classical and Dynamic Switching and Mixture Models , 2001 .

[32]  W. Wong,et al.  Real-Parameter Evolutionary Monte Carlo With Applications to Bayesian Mixture Models , 2001 .

[33]  M. Aitkin.  Likelihood and Bayesian analysis of mixtures , 2001 .

[34]  Jun S. Liu,et al.  Monte Carlo strategies in scientific computing , 2001 .

[35]  Adrian E. Raftery,et al.  Model-based clustering and data transformations for gene expression data , 2001, Bioinform..

[36]  G. Casella,et al.  Perfect samplers for mixtures of distributions , 2002 .

[37]  P. Green,et al.  Hidden Markov Models and Disease Mapping , 2002 .

[38]  J. Idier,et al.  Penalized Maximum Likelihood Estimator for Normal Mixtures , 2000 .

[39]  O. Cappé,et al.  Reversible jump, birth‐and‐death and more general continuous time Markov chain Monte Carlo samplers , 2003 .

[40]  C. Robert,et al.  Estimating Mixtures of Regressions , 2003 .

[41]  Christian P. Robert,et al.  Monte Carlo Statistical Methods , 2005, Springer Texts in Statistics.

[42]  Richard J Boys,et al.  A Bayesian Approach to DNA Sequence Segmentation , 2004, Biometrics.

[43]  Jean-Michel Marin,et al.  Bayesian Modelling and Inference on Mixtures of Distributions , 2005 .

[44]  Paul Fearnhead,et al.  Exact and efficient Bayesian inference for multiple changepoint problems , 2006, Stat. Comput..

[45]  Petros Dellaportas,et al.  Multivariate mixtures of normals with unknown number of components , 2006, Stat. Comput..

[46]  Ajay Jasra,et al.  Population-Based Reversible Jump Markov Chain Monte Carlo , 2007, 0711.0186.