Simultaneous perturbation stochastic approximation for clustering of a Gaussian mixture model under unknown but bounded disturbances

Multidimensional optimization holds a central role in many machine learning problems. When a model quality functional is measured with an almost arbitrary external noise, it makes sense to use randomized optimization techniques. This paper deals with the problem of clustering of a Gaussian mixture model under unknown but bounded disturbances. We introduce a stochastic approximation algorithm with randomly perturbed input (like SPSA) to solve this problem. The proposed method is appropriate for the online learning with streaming data, and it has a high speed of convergence. We study the conditions of the SPSA clustering algorithm applicability and show illustrative examples.

[1]  Olga Granichina,et al.  Control of Educational Processes Using SPSA , 2016, 2016 International Conference on Computational Science and Computational Intelligence (CSCI).

[2]  Adam Meyerson,et al.  Fast and Accurate k-means For Large Datasets , 2011, NIPS.

[3]  Mihaela van der Schaar,et al.  Ensemble Online Clustering through Decentralized Observations , 2014, 53rd IEEE Conference on Decision and Control.

[4]  N. Granichin Udc.  PROCEDURE OF STOCHASTIC APPROXIMATION WITH DISTURBANCES AT THE INPUT , 1992 .

[5]  O. Nelles,et al.  An Introduction to Optimization , 1996, IEEE Antennas and Propagation Magazine.

[6]  J. Blum.  Multidimensional Stochastic Approximation Methods , 1954 .

[7]  J. Spall.  Multivariate stochastic approximation using a simultaneous perturbation gradient approximation , 1992 .

[8]  D. Sculley,et al.  Web-scale k-means clustering , 2010, WWW '10.

[9]  J. Kiefer,et al.  Stochastic Estimation of the Maximum of a Regression Function , 1952 .

[10]  S. P. Lloyd,et al.  Least squares quantization in PCM , 1982, IEEE Trans. Inf. Theory.

[11]  Natalia O. Amelina,et al.  Simultaneous Perturbation Stochastic Approximation for Tracking Under Unknown but Bounded Disturbances , 2015, IEEE Transactions on Automatic Control.

[12]  O. Granichin.  Randomized Algorithms for Stochastic Approximation under Arbitrary Disturbances , 2002 .

[13]  Ali S. Hadi,et al.  Finding Groups in Data: An Introduction to Chster Analysis , 1991 .

[14]  H. Robbins.  A Stochastic Approximation Method , 1951 .

[15]  Peter J. Huber,et al.  Robust Statistics , 2005, Wiley Series in Probability and Statistics.

[16]  O. Granichin,et al.  A Randomized Stochastic Approximation Algorithm for Self-Learning , 2005 .

[17]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[18]  L. Hubert,et al.  Comparing partitions , 1985 .

[19]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[20]  Alexander T. Vakhitov,et al.  Algorithm for stochastic approximation with trial input perturbation in the nonstationary problem of optimization , 2009 .