Optimal Grid-Clustering: Towards Breaking the Curse of Dimensionality in High-Dimensional Clustering

Many applications require the clustering of large amounts of high-dimensional data. Most clustering algorithms, however, do not work e ectively and e ciently in highdimensional space, which is due to the so-called "curse of dimensionality". In addition, the high-dimensional data often contains a signi cant amount of noise which causes additional e ectiveness problems. In this paper, we review and compare the existing algorithms for clustering highdimensional data and show the impact of the curse of dimensionality on their e ectiveness and e ciency. The comparison reveals that condensation-based approaches (such as BIRCH or STING) are the most promising candidates for achieving the necessary e ciency, but it also shows that basically all condensation-based approaches have severe weaknesses with respect to their e ectiveness in highdimensional space. To overcome these problems, we develop a new clustering technique called OptiGrid which is based on constructing an optimal grid-partitioning of the data. The optimal grid-partitioning is determined by calculating the best partitioning hyperplanes for each dimension (if such a partitioning exists) using certain projections of the data. The advantages of our new approach are (1) it has a rm mathematical basis (2) it is by far more e ective than existing clustering algorithms for highdimensional data (3) it is very e cient even for large data sets of high dimensionality. To demonstrate the e ectiveness and e ciency of our new approach, we perform a series of experiments on a number of di erent data sets including real data sets from CAD and molecular biology. A comparison with one of the best known algorithms (BIRCH) shows the superiority of our new approach. Permission to copy without fee all or part of this material is granted provided that the copies are not made or distributed for direct commercial advantage, the VLDB copyright notice and the title of the publication and its date appear, and notice is given that copying is by permission of the Very Large Data Base Endowment. To copy otherwise, or to republish, requires a fee and/or special permission from the Endowment. Proceedings of the 25th VLDB Conference, Edinburgh, Scotland, 1999.