Inconsistency of Pitman-Yor process mixtures for the number of components

In many applications, a finite mixture is a natural model, but it can be difficult to choose an appropriate number of components. To circumvent this choice, investigators are increasingly turning to Dirichlet process mixtures (DPMs), and Pitman-Yor process mixtures (PYMs), more generally. While these models may be well-suited for Bayesian density estimation, many investigators are using them for inferences about the number of components, by considering the posterior on the number of components represented in the observed data. We show that this posterior is not consistent --- that is, on data from a finite mixture, it does not concentrate at the true number of components. This result applies to a large class of nonparametric mixtures, including DPMs and PYMs, over a wide variety of families of component distributions, including essentially all discrete families, as well as continuous exponential families satisfying mild regularity conditions (such as multivariate Gaussians).

[1]  S. Yakowitz,et al.  On the Identifiability of Finite Mixtures , 1968 .

[2]  D. Blackwell,et al.  Ferguson Distributions Via Polya Urn Schemes , 1973 .

[3]  T. Ferguson.  A Bayesian Analysis of Some Nonparametric Problems , 1973 .

[4]  C. Antoniak.  Mixtures of Dirichlet Processes with Applications to Bayesian Nonparametric Problems , 1974 .

[5]  丸山 徹.  Convex Analysisの二,三の進展について , 1977 .

[6]  P. Diaconis,et al.  Conjugate Priors for Exponential Families , 1979 .

[7]  D. Berry,et al.  Empirical Bayes Estimation of a Binomial Parameter Via Mixtures of Dirichlet Processes , 1979 .

[8]  T. Ferguson.  BAYESIAN DENSITY ESTIMATION BY MIXTURES OF NORMAL DISTRIBUTIONS , 1983 .

[9]  Albert Y. Lo,et al.  On a Class of Bayesian Nonparametric Estimates: I. Density Estimates , 1984 .

[10]  J. Henna.  On estimating of the number of constituents of a finite mixture of continuous distributions , 1985 .

[11]  B. Leroux.  Consistent estimation of a mixing distribution , 1992 .

[12]  Radford M. Neal.  Bayesian Mixture Modeling , 1992 .

[13]  M. West,et al.  Hyperparameter estimation in Dirichlet process mixture models , 1992 .

[14]  J. Pitman,et al.  Size-biased sampling of Poisson point processes and excursions , 1992 .

[15]  J. Hoffman-Jorgensen.  Probability with a View Towards Statistics, Volume II , 1994 .

[16]  J. Hoffmann-jorgensen,et al.  Probability with a View Toward Statistics , 1994 .

[17]  Jun S. Liu,et al.  The Collapsed Gibbs Sampler in Bayesian Computations with Applications to a Gene Regulation Problem , 1994 .

[18]  S. MacEachern.  Estimating normal means with a conjugate style dirichlet process prior , 1994 .

[19]  Walter R. Gilks,et al.  Bayesian model comparison via jump diffusions , 1995 .

[20]  M. Escobar,et al.  Bayesian Density Estimation and Inference Using Mixtures , 1995 .

[21]  T. Sapatinas.  Identifiability of mixtures of power-series distributions and related characterizations , 1995, Annals of the Institute of Statistical Mathematics.

[22]  J. Pitman.  Some developments of the Blackwell-MacQueen urn scheme , 1996 .

[23]  S. MacEachern,et al.  A semiparametric Bayesian model for randomised block designs , 1996 .

[24]  J. Pitman,et al.  The two-parameter Poisson-Dirichlet distribution derived from a stable subordinator , 1997 .

[25]  P. Green,et al.  Corrigendum: On Bayesian analysis of mixtures with an unknown number of components , 1997 .

[26]  S. MacEachern,et al.  Estimating mixture of dirichlet process models , 1998 .

[27]  Michael A. West,et al.  Computing Nonparametric Hierarchical Models , 1998 .

[28]  Steven N. MacEachern,et al.  Computational Methods for Mixture of Dirichlet Process Models , 1998 .

[29]  J. Ghosh,et al.  POSTERIOR CONSISTENCY OF DIRICHLET MIXTURES IN DENSITY ESTIMATION , 1999 .

[30]  J. Pitman,et al.  Prediction rules for exchangeable sequences related to species sampling ( , 2000 .

[31]  Radford M. Neal.  Markov Chain Sampling Methods for Dirichlet Process Mixture Models , 2000 .

[32]  M. Stephens.  Bayesian analysis of mixture models with an unknown number of components- an alternative to reversible jump methods , 2000 .

[33]  P. Donnelly,et al.  Inference of population structure using multilocus genotype data. , 2000, Genetics.

[34]  Lancelot F. James,et al.  Bayesian Model Selection in Finite Mixtures by Marginal Density Decompositions , 2001 .

[35]  Lancelot F. James,et al.  Gibbs Sampling Methods for Stick-Breaking Priors , 2001 .

[36]  A. V. D. Vaart,et al.  Entropies and rates of convergence for maximum likelihood and Bayes estimation for mixtures of normal densities , 2001 .

[37]  N. Levenberg,et al.  Function theory in several complex variables , 2001 .

[38]  P. Green,et al.  Modelling Heterogeneity With and Without the Dirichlet Process , 2001 .

[39]  Lancelot F. James,et al.  Consistent estimation of mixture complexity , 2001 .

[40]  Mario Medvedovic,et al.  Bayesian infinite mixture model based clustering of gene expression profiles , 2002, Bioinform..

[41]  E. Otranto,et al.  A NONPARAMETRIC BAYESIAN APPROACH TO DETECT THE NUMBER OF REGIMES IN MARKOV SWITCHING MODELS , 2002 .

[42]  Michael,et al.  On a Class of Bayesian Nonparametric Estimates : I . Density Estimates , 2008 .

[43]  Lancelot F. James,et al.  Generalized weighted Chinese restaurant processes for species sampling mixture models , 2003 .

[44]  J. Pitman,et al.  Exchangeable Gibbs partitions and Stirling triangles , 2004, math/0412494.

[45]  W. Kruijer.  Convergence Rates in Nonparametric Bayesian Density Estimation , 2004 .

[46]  Paul Fearnhead,et al.  Particle filters for mixture models with an unknown number of components , 2004, Stat. Comput..

[47]  H. Philippe,et al.  A Bayesian mixture model for across-site heterogeneities in the amino-acid replacement process. , 2004, Molecular biology and evolution.

[48]  A. W. Knapp.  Basic Real Analysis , 2005 .

[49]  J. Henna.  Estimation of the number of components of finite mixtures of multivariate distributions , 2005 .

[50]  S. Walker,et al.  On Consistency of Nonparametric Normal Mixtures for Bayesian Density Estimation , 2005 .

[51]  P. Arctander,et al.  Regional genetic structuring and evolutionary history of the impala Aepyceros melampus. , 2006, The Journal of heredity.

[52]  P. Müller,et al.  Bayesian inference for gene expression and proteomics , 2006 .

[53]  Yee Whye Teh,et al.  Bayesian multi-population haplotype inference via a hierarchical dirichlet process mixture , 2006, ICML.

[54]  T. N. Sriram,et al.  Robust Estimation of Mixture Complexity , 2006 .

[55]  Michael A. West,et al.  Hierarchical priors and mixture models, with applications in regression and density estimation , 2006 .

[56]  J. Pella,et al.  The Gibbs and splitmerge sampler for population mixture analysis from genetic data with incomplete baselines , 2006 .

[57]  J. Pitman.  Combinatorial Stochastic Processes , 2006 .

[58]  S. Ghosal.  Semiparametric Accelerated Failure Time Models for Censored Data , 2006 .

[59]  P. Müller,et al.  10 Model-Based Clustering for Expression Data via a Dirichlet Process Mixture Model , 2006 .

[60]  A. Fearnside.  Bayesian analysis of finite mixture distributions using the allocation sampler , 2007 .

[61]  E. González,et al.  Relative role of life-history traits and historical factors in shaping genetic population structure of sardines (Sardina pilchardus) , 2007, BMC Evolutionary Biology.

[62]  T. N. Sriram,et al.  Robust estimation of mixture complexity for count data , 2007, Comput. Stat. Data Anal..

[63]  J. Huelsenbeck,et al.  Inference of Population Structure Under a Dirichlet Process Model , 2007, Genetics.

[64]  Alan S. Willsky,et al.  Hierarchical Dirichlet processes for tracking maneuvering targets , 2007, 2007 10th International Conference on Information Fusion.

[65]  Agostino Nobile,et al.  Bayesian finite mixtures with an unknown number of components: The allocation sampler , 2007, Stat. Comput..

[66]  M. Stephens,et al.  Inference of population structure using multilocus genotype data: dominant markers and null alleles , 2007, Molecular ecology notes.

[67]  S. Walker,et al.  On rates of convergence for posterior distributions in infinite-dimensional models , 2007, 0708.1892.

[68]  A. V. D. Vaart,et al.  Posterior convergence rates of Dirichlet mixtures at smooth densities , 2007, 0708.1885.

[69]  Ramsés H. Mena,et al.  Bayesian Nonparametric Estimation of the Probability of Discovering New Species , 2007 .

[70]  S. Ghosal,et al.  Posterior consistency of Dirichlet mixtures for estimating a transition density , 2007 .

[71]  S. Walker,et al.  Bayesian nonparametric estimators derived from conditional Gibbs structures , 2008, 0808.2863.

[72]  C. Richards,et al.  Genetic diversity and population structure in Malus sieversii, a wild progenitor species of domesticated apple , 2009, Tree Genetics & Genomes.

[73]  Lancelot F. James.  Large sample asymptotics for the two-parameter Poisson–Dirichlet process , 2007, 0708.4294.

[74]  A. Pievatolo,et al.  A comparison of nonparametric priors in hierarchical mixture modelling for AFT regression , 2009 .

[75]  S. Ghosal,et al.  2 The Dirichlet process , related priors and posterior asymptotics , 2009 .

[76]  Haofeng Chen,et al.  Tracing the geographic origins of major avocado cultivars. , 2008, The Journal of heredity.

[77]  A. Leaché,et al.  Bayesian species delimitation in West African forest geckos (Hemidactylus fasciatus) , 2010, Proceedings of the Royal Society B: Biological Sciences.

[78]  Gun Ho Jang,et al.  POSTERIOR CONSISTENCY OF SPECIES SAMPLING PRIORS , 2010 .

[79]  Mitsuo Morita,et al.  Characterization of a Bayesian genetic clustering algorithm based on a Dirichlet process prior and comparison among Bayesian clustering methods , 2011, BMC Bioinformatics.

[80]  S. Ghosal.  Bayesian Nonparametrics: The Dirichlet process, related priors and posterior asymptotics , 2010 .

[81]  Hongbin Zha,et al.  CDP Mixture Models for Data Clustering , 2010, 2010 20th International Conference on Pattern Recognition.

[82]  Yuefeng Wu,et al.  The L1-consistency of Dirichlet mixtures in multivariate Bayesian density estimation , 2010, J. Multivar. Anal..

[83]  A. V. D. Vaart,et al.  Adaptive Bayesian density estimation with location-scale mixtures , 2010 .

[84]  D. Dunson,et al.  Nonparametric Bayesian density estimation on manifolds with applications to planar shapes. , 2010, Biometrika.

[85]  K. Mengersen,et al.  Asymptotic behaviour of the posterior distribution in overfitted mixture models , 2011 .

[86]  Catia Scricciolo.  Adaptive Bayesian density estimation using Pitman-Yor or normalized inverse-Gaussian process kernel mixtures , 2012, 1210.8094.

[87]  Matthew T. Harrison,et al.  A simple example of Dirichlet process mixture inconsistency for the number of components , 2013, NIPS.

[88]  David B. Dunson,et al.  Posterior consistency in conditional distribution estimation , 2013, J. Multivar. Anal..

[89]  X. Nguyen.  Convergence of latent mixing measures in finite and infinite mixture models , 2011, 1109.3250.