Neighborhood Graphs for Estimation of Density Functionals.

Functionals of densities play a fundamental role in statistics, signal processing, machine learning, information theory and related fields. This class of functionals includes entropy, divergence and mutual information measures of densities, intrinsic dimension of data embedded in manifolds, and minimum volume sets of densities. k-nearest neighbor (k-NN) graph based estimators are widely used for the estimation of these functionals. While several consistent k-NN estimators have been previously proposed for estimating these functionals, general results on rates of convergence of these estimators and confidence intervals on the estimated functional are not available. Since the rate of convergence relates the number of samples to the performance of the estimator, convergence rates have great practical utility. In this thesis, a new class of estimators based on bipartite k-nearest neighbor graphs is proposed for estimating functionals of probability density functions. This class includes entropy and divergence estimators, intrinsic dimension estimators and estimates of p-values for testing membership of data in minimum volume sets. For this class of estimators, large sample theory is used to characterize performance of the estimators. Specifically, large sample expressions for estimator bias and variance is derived and a central limit theorem for the distribution of the estimators is established. This theory is applied to accurately estimate functionals of interest by optimizing the mean squared error over free parameters, e.g. the number of neighbors k, and obtaining confidence intervals on the estimated functional by invoking the central limit theorem. Furthermore, this theory provides significant insight into the statistical