Disentangling Gaussians

The Gaussian mixture model (GMM) is one of the oldest and most widely-used statistical models. It is comprised of a weighted combination of heterogeneous Gaussian sources. As a simple one–dimensional example, consider measurements of heights of adults in a certain population, where the distribution of heights can be closely approximated as a mixture of two univariate Gaussians, one for males and one for females [3]. Can one recover the parameters of the Gaussians from unlabeled height measurements alone (with no gender information)? This paper focuses on the case where the mixture consists of a small but unknown number of Gaussians that may overlap—the combined density may even have a single peak, as in the height example, and the dimensionality may be high. Much of the previous work on this problem attempts to learn the parameters through clustering, and consequently needs to make a strong separation assumption on the components in the mixture. The primary contribution of our research is to avoid this assumption by instead basing our learning algorithm upon the algebraic structure of the mixture. Our algorithm succeeds even if the components overlap almost entirely, in which case accurate clustering is no longer possible. We give a simple notion of “condition number” of a GMM which characterizes its complexity up to polynomial factors. Generally speaking, the conclusion is that the statistical complexity and computational complexity of this general problem is in every way polynomial except for the dependence on the number of Gaussians, which is necessarily exponential. Statisticians have long known that from random samples from a GMM it is possible to identify the Gaussians in the limit—one can eventually recover to arbitrary precision each subpopulation’s mean, variance, and proportion, given sufficiently many examples [14]. However, their analysis provides no bounds on convergence rate—it might be exponentially slow even for two Gaussians in one dimension. Moreover, heuristics in widespread use, such as the EM algorithm, suf-