Bandit-Based Monte Carlo Optimization for Nearest Neighbors

The celebrated Monte Carlo method estimates an expensive-to-compute quantity by random sampling. Bandit-based Monte Carlo optimization is a general technique for computing the minimum of many such expensive-to-compute quantities by adaptive random sampling. The technique converts an optimization problem into a statistical estimation problem which is then solved via multi-armed bandits. We apply this technique to solve the problem of high-dimensional <inline-formula> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula>-nearest neighbors, developing an algorithm which we prove is able to identify exact nearest neighbors with high probability. We show that under regularity assumptions on a dataset of <inline-formula> <tex-math notation="LaTeX">$n$ </tex-math></inline-formula> points in <inline-formula> <tex-math notation="LaTeX">$d$ </tex-math></inline-formula>-dimensional space, the complexity of our algorithm scales logarithmically with the dimension of the data as <inline-formula> <tex-math notation="LaTeX">$O\left({(n+d)\log ^{2} \frac {nd}{\delta }}\right)$ </tex-math></inline-formula> for error probability <inline-formula> <tex-math notation="LaTeX">$\delta $ </tex-math></inline-formula>, rather than linearly as in exact computation requiring <inline-formula> <tex-math notation="LaTeX">$O(nd)$ </tex-math></inline-formula>. We corroborate our theoretical results with numerical simulations, showing that our algorithm outperforms both exact computation and state-of-the-art algorithms such as kGraph, NGT, and LSH on real datasets.