Learning mixtures of arbitrary gaussians

Mixtures of gaussian (or normal) distributions arise in a variety of application areas. Many techniques have been proposed for the task of finding the component gaussians given samples from the mixture, such as the EM algorithm, a local-search heuristic from Dempster, Laird and Rubin~(1977). However, such heuristics are known to require time exponential in the dimension (i.e., number of variables) in the worst case, even when the number of components is $2$. This paper presents the first algorithm that provably learns the component gaussians in time that is polynomial in the dimension. The gaussians may have arbitrary shape provided they satisfy a “nondegeneracy” condition, which requires their high-probability regions to be not “too close” together.

[1]  A. Prékopa.  Logarithmic concave measures with applications to stochastic programming , 1971 .

[2]  L. Leindler.  On a Certain Converse of Hölder’s Inequality , 1972 .

[3]  W. Hoeffding.  Probability Inequalities for sums of Bounded Random Variables , 1963 .

[4]  R. Redner,et al.  Mixture densities, maximum likelihood, and the EM algorithm , 1984 .

[5]  Sanjoy Dasgupta,et al.  Learning mixtures of Gaussians , 1999, 40th Annual Symposium on Foundations of Computer Science (Cat. No.99CB37039).

[6]  Sudipto Guha,et al.  A constant-factor approximation algorithm for the k-median problem (extended abstract) , 1999, STOC '99.

[7]  M. Rudelson.  Random Vectors in the Isotropic Position , 1996, math/9608208.

[8]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[9]  B. Lindsay.  Mixture models : theory, geometry, and applications , 1995 .

[10]  Yishay Mansour,et al.  Estimating a mixture of two product distributions , 1999, COLT '99.

[11]  A. Prékopa.  On logarithmic concave measures and functions , 1973 .

[12]  A. F. Smith,et al.  Statistical analysis of finite mixture distributions , 1986 .

[13]  W. B. Johnson,et al.  Extensions of Lipschitz mappings into Hilbert space , 1984 .

[14]  Peter Kall,et al.  Stochastic Programming , 1995 .

[15]  Sudipto Guha,et al.  A constant-factor approximation algorithm for the k-median problem (extended abstract) , 1999, STOC '99.

[16]  J. Bourgain.  Random Points in Isotropic Convex Sets , 1998 .

[17]  Robin Sibson,et al.  What is projection pursuit , 1987 .

[18]  Noga Alon,et al.  The Probabilistic Method , 2015, Fundamentals of Ramsey Theory.

[19]  Ravi Kannan,et al.  Sampling according to the multivariate normal density , 1996, Proceedings of 37th Conference on Foundations of Computer Science.