Learning via Hilbert Space Embedding of Distributions

In this thesis, we propose a framework for learning based on Hilbert space embedding of distributions. By embedding the distributions via the kernel mean map, we are able to compare distributions by computing their distance in the reproducing Hilbert spaces. We show that learning via mean map leads to both good generalization ability and finite sample convergence. Using this distance between distributions, we are able to tackle a wide range of learning problems under a common framework. Very often this new view leads to simpler and more effective algorithms in various learning problems. In particular, this thesis focuses on a measure of dependence based on the mean map, and we show that it can tackle a raft of learning problems from which we singled out four concrete examples and discussed in details in the thesis: • Independence measure and test for structured and heterogeneous data. • Feature selection via dependence for supervised learning scenario. • Clustering via dependence with additional metric on labels. • Dimensionality reduction via dependence with side information. We also show that learning via Hilbert space embedding/dependence subsumes many existing algorithms as special cases. By elucidating the differences and connections of these algorithms, we are able to provide useful guidelines for practitioners in various applications. This embedding approach for distribution analysis offers us a principled drop-in replacement for information theoretic approaches. We believe it will have wide applications in near future.