In this paper, we investigate how to scale hierarchical clustering methods (such as OPTICS) to extremely large databases by utilizing data compression methods (such as BIRCH or random sampling). We propose a three step procedure: 1) compress the data into suitable representative objects; 2) apply the hierarchical clustering algorithm only to these objects; 3) recover the clustering structure for the whole data set, based on the result for the compressed data. The key issue in this approach is to design compressed data items such that not only a hierarchical clustering algorithm can be applied, but also that they contain enough information to infer the clustering structure of the original data set in the third step. This is crucial because the results of hierarchical clustering algorithms, when applied naively to a random sample or to the clustering features (CFs) generated by BIRCH, deteriorate rapidly for higher compression rates. This is due to three key problems, which we identify. To solve these problems, we propose an efficient post-processing step and the concept of a Data Bubble as a special kind of compressed data item. Applying OPTICS to these Data Bubbles allows us to recover a very accurate approximation of the clustering structure of a large data set even for very high compression rates. A comprehensive performance and quality evaluation shows that we only trade very little quality of the clustering result for a great increase in performance.
[1]
Robin Sibson,et al.
SLINK: An Optimally Efficient Algorithm for the Single-Link Cluster Method
,
1973,
Comput. J..
[2]
Theodore Johnson,et al.
Squashing flat files flatter
,
1999,
KDD '99.
[3]
Paul S. Bradley,et al.
Scaling Clustering Algorithms to Large Databases
,
1998,
KDD.
[4]
J. MacQueen.
Some methods for classification and analysis of multivariate observations
,
1967
.
[5]
Ali S. Hadi,et al.
Finding Groups in Data: An Introduction to Chster Analysis
,
1991
.
[6]
Hans-Peter Kriegel,et al.
Fast Hierarchical Clustering Based on Compressed Data and OPTICS
,
2000,
PKDD.
[7]
Hans-Peter Kriegel,et al.
A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise
,
1996,
KDD.
[8]
Anil K. Jain,et al.
Algorithms for Clustering Data
,
1988
.
[9]
Tian Zhang,et al.
BIRCH: an efficient data clustering method for very large databases
,
1996,
SIGMOD '96.
[10]
Hans-Peter Kriegel,et al.
OPTICS: ordering points to identify the clustering structure
,
1999,
SIGMOD '99.