Learning mixtures of distributions

This thesis studies the problem of learning mixtures of distributions, a natural formalization of clustering. A mixture of distributions is a collection of distributions D = {D1, ... DT}, and mixing weights, {w1,..., wT} such that Σiwi = 1. A sample from a mixture is generated by choosing i with probability wi and choosing a sample from distribution Di. Given samples from a mixture of distributions, the problem of learning the mixture is that of finding the parameters of the distributions comprising D and grouping the samples according to source distribution. A common theoretical framework for addressing the problem also assumes that we are given a separation condition, which is a promise that any two distributions in the mixture are sufficiently different. In this thesis, we study three aspects of the problem. First, in Chapter 3, we focus on optimizing the separation condition while learning mixtures of distributions. The most common algorithms in practice are singular value decomposition based algorithms, which work when the separation is Θ pswmin p , where σ is the maximum directional standard deviation of any distribution in the mixture, and wmin is the minimum mixing weight. We show an algorithm which successfully learns mixtures of distributions with a separation condition that depends only logarithmically on the skewed mixing weights. In particular, it succeeds for a separation between the centers that is Θ pTlogLp , where T is the number of distributions, and Λ is polynomial in T and the imbalance in the mixing weights. We require that the distance between the centers be spread across Θ(T log Λ) coordinates. In addition, we show that if every vector in the subspace spanned by the centers has a small projection, of the order of 1TlogL on each coordinate vector, then, our algorithm succeeds for a separation of only O ps*TlogL p , where σ* is the maximum directional standard deviation in the space containing the centers. Our algorithm works for Binary Product Distributions and Axis-Aligned Gaussians. The spreading condition above is implied by the separation condition for binary product distributions, and is necessary for algorithms that rely on linear correlations. Motivated by the application in population genetics, in Chapter 4, we study the sample complexity of learning mixtures of binary product distributions. In this thesis, we take a step towards learning mixtures of binary product distributions with optimal sample complexity by providing an algorithm which learns a mixture of two binary product distributions with uniform mixing weights and low sample complexity. Our algorithm clusters all the samples correctly with high probability, so long as d(μ1, μ 2) the square of the Euclidean distance between the centers of distributions is at least polylogarithmic in s, the number of samples and the following trade-off holds between the separation and the number of samples: sdd2pm1,m 2p≥adnlogslogpnsp for some constant a. Finally, in Chapter 5, we study the problem of learning mixtures of heavy-tailed product distributions. To this end, we provide an embedding from R n to {0, 1}n', which maps random samples from distributions with medians that are far apart to random samples from distributions on {0, 1}n', with centers that are far apart. The main application of our embedding is in designing an algorithm for learning mixtures of heavy-tailed distributions. We provide a polynomial-time algorithm, which learns mixtures of general product distributions, as long as the distribution of each coordinate satisfies two properties: symmetry about the median and ¾-radius upper-bounded by R. The separation required by our algorithm to correctly classify a 1–δ fraction of the samples is that the distance between the medians of any two distributions in the mixture should be O pRTlogL+R TlogTd p , and this distance should be spread across O(T log Λ + T log Td ) coordinates. A second application of our embedding is in designing algorithms for learning mixtures of distributions with finite variance, which work under a separation requirement of O ps*TlogL p and a spreading requirement of O(T log Λ + T log Td ). This algorithm does not require the more stringent spreading condition needed by the algorithm which offers similar guarantees in Chapter 3.