Entropy-Based Incremental Variational Bayes Learning of Gaussian Mixtures

Variational approaches to density estimation and pattern recognition using Gaussian mixture models can be used to learn the model and optimize its complexity simultaneously. In this brief, we develop an incremental entropy-based variational learning scheme that does not require any kind of initialization. The key element of the proposal is to exploit the incremental learning approach to perform model selection through efficient iteration over the variational Bayes optimization step in a way that the number of splits is minimized. The method starts with just one component and adds new components iteratively by splitting the worst fitted kernel in terms of evaluating its entropy. Our experimental results, on synthetic and real data sets show the effectiveness of the approach outperforming other state-of-the-art incremental component learners.

[1]  Dirk Husmeier,et al.  The Bayesian Evidence Scheme for Regularizing Probability-Density Estimating Neural Networks , 2000, Neural Computation.

[2]  Kap Luk Chan,et al.  Learning Multivariate Gaussian Mixtures with the Reversible Jump MCMC Algorithm , 2004 .

[3]  Aristidis Likas,et al.  Unsupervised Learning of Gaussian Mixtures Based on Variational Component Splitting , 2007, IEEE Transactions on Neural Networks.

[4]  Anil K. Jain,et al.  Unsupervised selection and estimation of finite mixture models , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[5]  P. Green,et al.  On Bayesian Analysis of Mixtures with an Unknown Number of Components (with discussion) , 1997 .

[6]  Adrian Corduneanu,et al.  Variational Bayesian Model Selection for Mixture Distributions , 2001 .

[7]  George E. P. Box,et al.  Bayesian Inference in Statistical Analysis: Box/Bayesian , 1992 .

[8]  Adrian G. Bors,et al.  Minimal Topology for a Radial Basis Functions Neural Network for Pattern Classification , 1994 .

[9]  Petros Dellaportas,et al.  Multivariate mixtures of normals with unknown number of components , 2006, Stat. Comput..

[10]  Juan Manuel Sáez,et al.  Learning Gaussian Mixture Models With Entropy-Based Criteria , 2009, IEEE Transactions on Neural Networks.

[11]  Hagai Attias,et al.  Inferring Parameters and Structure of Latent Variable Models by Variational Bayes , 1999, UAI.

[12]  G. Arfken.  Mathematical Methods for Physicists , 1967 .

[13]  José M. N. Leitão,et al.  On Fitting Mixture Models , 1999, EMMCVPR.

[14]  Zoubin Ghahramani,et al.  Variational Inference for Bayesian Mixtures of Factor Analysers , 1999, NIPS.

[15]  L. Pronzato,et al.  A class of Rényi information estimators for multidimensional densities , 2008, 0810.5302.

[16]  Francisco Escolano,et al.  Entropy-Based Variational Scheme for Fast Bayes Learning of Gaussian Mixtures , 2010, SSPR/SPR.

[17]  Anil K. Jain,et al.  Unsupervised Learning of Finite Mixture Models , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  David J. C. Mackay,et al.  Introduction to Monte Carlo Methods , 1998, Learning in Graphical Models.

[19]  A. F. Smith,et al.  Statistical analysis of finite mixture distributions , 1986 .

[20]  A. Hero,et al.  Estimation of Renyi information divergence via pruned minimal spanning trees , 1999, Proceedings of the IEEE Signal Processing Workshop on Higher-Order Statistics. SPW-HOS '99.

[21]  A. F. Smith,et al.  Statistical analysis of finite mixture distributions , 1986 .

[22]  Adrian G. Bors,et al.  Blind Source Separation USing Variational Expectation-Maximization Algorithm , 2003, CAIP.

[23]  Anil K. Jain,et al.  Statistical Pattern Recognition: A Review , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[24]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[25]  N. Nasios,et al.  Variational learning for Gaussian mixture models , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).