Online variational learning of finite Dirichlet mixture models

In this paper, we present an online variational inference algorithm for finite Dirichlet mixture models learning. Online algorithms allow data points to be processed one at a time, which is important for real-time applications, and also where large scale data sets are involved so that batch processing of all data points at once becomes infeasible. By adopting the variational Bayes framework in an online manner, all the involved parameters and the model complexity (i.e. the number of components) of the Dirichlet mixture model can be estimated simultaneously in a closed form. The proposed algorithm is validated through both synthetic data sets and a challenging real-world application namely video background subtraction.

[1]  Neil D. Lawrence,et al.  Mixture Representations for Inference and Learning in Boltzmann Machines , 1998, UAI.

[2]  Arne Leijon,et al.  Bayesian Estimation of Beta Mixture Models with Variational Inference , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  G. Parisi,et al.  Statistical Field Theory , 1988 .

[4]  Nizar Bouguila,et al.  Unsupervised learning of a finite mixture model based on the Dirichlet distribution and its application , 2004, IEEE Transactions on Image Processing.

[5]  Ferdinand van der Heijden,et al.  Efficient adaptive density estimation per image pixel for the task of background subtraction , 2006, Pattern Recognit. Lett..

[6]  W. Eric L. Grimson,et al.  Adaptive background mixture models for real-time tracking , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[7]  W. Eric L. Grimson,et al.  Learning Patterns of Activity Using Real-Time Tracking , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Bo Wang,et al.  Inadequacy of interval estimates corresponding to variational Bayesian approximations , 2005, AISTATS.

[9]  Léon Bottou,et al.  On-line learning and stochastic approximations , 1999 .

[10]  Neil D. Lawrence,et al.  Approximating Posterior Distributions in Belief Networks Using Mixtures , 1997, NIPS.

[11]  Edward H. Adelson,et al.  Representing moving images with layers , 1994, IEEE Trans. Image Process..

[12]  Christoph Stiller,et al.  Object-based estimation of dense motion fields , 1997, IEEE Trans. Image Process..

[13]  Nizar Bouguila,et al.  Unsupervised learning of a finite discrete mixture: Applications to texture modeling and image databases summarization , 2007, J. Vis. Commun. Image Represent..

[14]  Bo Wang,et al.  Convergence and Asymptotic Normality of Variational Bayesian Approximations for Expon , 2004, UAI.

[15]  Nizar Bouguila,et al.  Finite Generalized Gaussian Mixture Modeling and Applications to Image and Video Foreground Segmentation , 2007, Fourth Canadian Conference on Computer and Robot Vision (CRV '07).

[16]  Nizar Bouguila,et al.  Unsupervised selection of a finite Dirichlet mixture model: an MML-based approach , 2006, IEEE Transactions on Knowledge and Data Engineering.

[17]  A. Hamza,et al.  Software modules categorization through likelihood and bayesian analysis of finite dirichlet mixtures , 2010 .

[18]  Nizar Bouguila,et al.  Using unsupervised learning of a finite Dirichlet mixture model to improve pattern recognition applications , 2005, Pattern Recognit. Lett..

[19]  Massimo Piccardi,et al.  Background subtraction techniques: a review , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[20]  Nizar Bouguila,et al.  MML-Based Approach for Finite Dirichlet Mixture Estimation and Selection , 2005, MLDM.

[21]  P. Diaconis,et al.  Conjugate Priors for Exponential Families , 1979 .

[22]  Satoshi Morinaga,et al.  Online heterogeneous mixture modeling with marginal and copula selection , 2011, KDD.

[23]  Nizar Bouguila,et al.  A probabilistic approach for shadows modeling and detection , 2005, IEEE International Conference on Image Processing 2005.

[24]  Nizar Bouguila,et al.  Online clustering via finite mixtures of Dirichlet and minimum message length , 2006, Eng. Appl. Artif. Intell..

[25]  N. Nasios,et al.  Variational learning for Gaussian mixture models , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[26]  Dar-Shyang Lee,et al.  Effective Gaussian mixture learning for video background subtraction , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Adrian Corduneanu,et al.  Variational Bayesian Model Selection for Mixture Distributions , 2001 .

[28]  Ferdinand van der Heijden,et al.  Recursive unsupervised learning of finite mixture models , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  D. Ziou,et al.  Ieee Workshop on Machine Learning for Signal Processing Improving Content Based Image Retrieval Systems Using Finite M U Lt I N 0 M I a L D I Rich Let M I Xtu R E , 2022 .

[30]  Nizar Bouguila,et al.  Image and Video Segmentation by Combining Unsupervised Generalized Gaussian Mixture Modeling and Feature Selection , 2010, IEEE Transactions on Circuits and Systems for Video Technology.

[31]  Chong Wang,et al.  Online Variational Inference for the Hierarchical Dirichlet Process , 2011, AISTATS.

[32]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[33]  Hagai Attias,et al.  A Variational Bayesian Framework for Graphical Models , 1999 .

[34]  Nizar Bouguila,et al.  On Fitting Finite Dirichlet Mixture Using ECM and MML , 2005, ICAPR.

[35]  Mark W. Woolrich,et al.  Variational bayes inference of spatial mixture models for segmentation , 2006, IEEE Transactions on Medical Imaging.

[36]  Michael I. Jordan,et al.  An Introduction to Variational Methods for Graphical Models , 1999, Machine Learning.

[37]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[38]  Harold J. Kushner,et al.  Stochastic Approximation Algorithms and Applications , 1997, Applications of Mathematics.

[39]  King Ngi Ngan,et al.  Automatic segmentation of moving objects for video object plane generation , 1998, IEEE Trans. Circuits Syst. Video Technol..

[40]  Shun-ichi Amari,et al.  Natural Gradient Works Efficiently in Learning , 1998, Neural Computation.

[41]  Christian P. Robert,et al.  Monte Carlo Statistical Methods , 2005, Springer Texts in Statistics.

[42]  Francis R. Bach,et al.  Online Learning for Latent Dirichlet Allocation , 2010, NIPS.

[43]  Masa-aki Sato,et al.  Online Model Selection Based on the Variational Bayes , 2001, Neural Computation.

[44]  Christian P. Robert,et al.  The Bayesian choice , 1994 .