Model Selection for Gaussian Mixture Models

This paper is concerned with an important issue in finite mixture modelling, the selection of the number of mixing components. We propose a new penalized likelihood method for model selection of finite multivariate Gaussian mixture models. The proposed method is shown to be statistically consistent in determining of the number of components. A modified EM algorithm is developed to simultaneously select the number of components and to estimate the mixing weights, i.e. the mixing probabilities, and unknown parameters of Gaussian distributions. Simulations and a real data analysis are presented to illustrate the performance of the proposed method.

[1]  Surajit Ray,et al.  Model selection in high dimensions: a quadratic‐risk‐based approach , 2006, math/0611544.

[2]  Jiahua Chen.  Optimal Rate of Convergence for Finite Mixture Models , 1995 .

[3]  T. N. Sriram,et al.  Robust Estimation of Mixture Complexity , 2006 .

[4]  Volker Tresp,et al.  Averaging, maximum penalized likelihood and Bayesian estimation for improving Gaussian mixture probability density estimates , 1998, IEEE Trans. Neural Networks.

[5]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[6]  A. Tsybakov,et al.  SPADES AND MIXTURE MODELS , 2009, 0901.2044.

[7]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[8]  E. Gassiat,et al.  Testing in locally conic models, and application to mixture models , 1997 .

[9]  Adrian Corduneanu,et al.  Variational Bayesian Model Selection for Mixture Distributions , 2001 .

[10]  Naonori Ueda,et al.  Deterministic annealing EM algorithm , 1998, Neural Networks.

[11]  Jiahua Chen,et al.  Order Selection in Finite Mixture Models With a Nonsmooth Penalty , 2008 .

[12]  L. Wasserman,et al.  Practical Bayesian Density Estimation Using Mixtures of Normals , 1997 .

[13]  John D. Kalbfleisch,et al.  Penalized minimum‐distance estimates in finite mixture models , 1996 .

[14]  Runze Li,et al.  Tuning parameter selectors for the smoothly clipped absolute deviation method. , 2007, Biometrika.

[15]  Geoffrey E. Hinton,et al.  SMEM Algorithm for Mixture Models , 1998, Neural Computation.

[16]  E. Gassiat,et al.  Testing the order of a model using locally conic parametrization : population mixtures and stationary ARMA processes , 1999 .

[17]  X. Nguyen.  Convergence of latent mixing measures in finite and infinite mixture models , 2011, 1109.3250.

[18]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[19]  Lancelot F. James,et al.  Consistent estimation of mixture complexity , 2001 .

[20]  Ferdinand van der Heijden,et al.  Recursive unsupervised learning of finite mixture models , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Geoffrey E. Hinton,et al.  The EM algorithm for mixtures of factor analyzers , 1996 .

[22]  P. Deb.  Finite Mixture Models , 2008 .

[23]  Anil K. Jain,et al.  Unsupervised Learning of Finite Mixture Models , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[24]  Matthew Brand,et al.  Structure Learning in Conditional Probability Models via an Entropic Prior and Parameter Extinction , 1999, Neural Computation.

[25]  Jianqing Fan,et al.  Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties , 2001 .

[26]  Nhat Ho,et al.  Identifiability and optimal rates of convergence for parameters of multiple types in finite mixtures , 2015, 1501.02497.

[27]  B. Lindsay.  Mixture models : theory, geometry, and applications , 1995 .

[28]  B. Leroux.  Consistent estimation of a mixing distribution , 1992 .