Fast, accurate spectral clustering using locally linear landmarks

For problems of image or video segmentation, where clusters have a complex structure, a leading method is spectral clustering. It works by encoding the similarity between pairs of points into an affinity matrix and applying k-means in its low-order eigenspace, where the clustering structure is enhanced. When the number of points is large, an approximation is necessary to limit the runtime even if the affinity matrix is sparse. This is commonly done with the Nystrom formula, where one solves an eigenproblem using affinities between a subset of the data points (landmarks) and then estimates the eigenvectors over the entire data by interpolation. In practice, this can still require many landmarks to achieve reasonably accurate solutions, and applies only for explicitly defined affinity kernels. In this paper we propose two ideas: the Locally Linear Landmarks technique, where one solves a reduced spectral problem over landmarks that involves the entire, original affinity matrix; and a fast, good initialization for k-means. We show both approximation error and runtime are considerably reduced, even though fewer landmarks are used. We apply it to spectral clustering and to several variants of it that involve complex affinities: constrained clustering, affinity aggregation, neighborhood graphs based on tree ensembles, and video segmentation.