Finding Clusters of Different Sizes, Shapes, and Densities in Noisy, High Dimensional Data

Finding clusters in data, especially high dimensional data, is challenging when the clusters are of widely diering shapes, sizes, and densities, and when the data contains noise and outliers. We present a novel clustering technique that addresses these issues. Our algorithm rst nds the nearest neighbors of each data point and then redenes the similarity between pairs of points in terms of how many nearest neighbors the two points share. Using this denition of similarity, our algorithm identies core points and then builds clusters around the core points. The use of a shared nearest neighbor denition of similarity alleviates problems with varying densities and high dimensionality, while the use of core points handles problems with shape and size. While our algorithm can nd the \dense" clusters that other clustering algorithms nd, it also nds clusters that these approaches overlook, i.e., clusters of low or medium density which represent relatively uniform regions \surrounded" by non-uniform or higher density areas. We experimentally show that our algorithm performs better than traditional methods (e.g., K-means, DBSCAN, CURE) on a variety of data sets: KDD Cup '99 network intrusion data, NASA Earth science time series data, two-dimensional point sets, and documents. The run-time complexity of our technique is O(n 2 ) if the similarity matrix has to be constructed. However, we discuss a number of optimizations that allow the algorithm to handle large data sets ecien tly.