Clustering via Similarity Functions : Theoretical Foundations and Algorithms ∗

Problems of clustering data from pairwise similarity information arise in many different fields. Yet the question of which algorithm is best to use under what conditions, and how good a notion of similarity does one need in order to cluster accurately remains poorly understood. In this work we propose a new general framework for analyzing clustering from similarity information that directly addresses this question of what properties of a similarity measure are sufficient to cluster accurately and by what kinds of algorithms. We show that in our framework a wide variety of interesting learning-theoretic and game-theoretic properties, including properties motivated by mathematical biology, can be used to cluster well, and we design new efficient algorithms that are able to take advantage of them. We consider two natural clustering objectives: (a) list clustering, where the algorithm’s goal is to produce a small list of clusterings such that at least one of them is approximately correct, and (b) hierarchical clustering, where the algorithm’s goal is to produce a hierarchy such that desired clustering is some pruning of this tree (which a user could navigate). We develop a notion of the clustering complexity of a given property, analogous to notions of capacity in learning theory, that characterizes information-theoretic usefulness for clustering. We analyze this quantity for a wide range of properties, giving tight upper and lower bounds. We also show how our algorithms can be extended to the inductive case, i.e., by using just a constant-sized sample, as in property testing. While our algorithms for this setting remain very efficient, proving correctness requires subtle analysis based on regularity-type results. Our framework can be viewed as an analog for clustering of discriminative models for supervised classification (i.e., the Statistical Learning Theory framework and the PAC learning model), where our goal is to cluster accurately given a property or relation the similarity function is believed to satisfy with respect to the ground truth clustering. More specifically our framework is analogous to that of data-dependent concept classes in supervised learning, where conditions such as the large margin property have been central in the analysis of kernel methods.